10727193 2003-09-22 16:49 +0400 /21 rader/ =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
Importerad: 2003-09-22 19:33 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Externa svar till: alienhard@mail.ru
Mottagare: Bugtraq (import) <29129>
Ärende: base64
------------------------------------------------------------
From: =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
To: bugtraq@securityfocus.com
Message-ID: <E1A1Q8d-0004A3-00.alienhard-mail-ru@f9.mail.ru>
Consider we decoding data which contains padding character ('=') at
the unexpected place. What we should do with such data? The
specification of base64 decoding does not tell us what we MUST or
even MAY do with such data... So, we can do anything we like to do:
1. threat padding character as end of the encoded data 2. ignore
padding character 3. decode padding character as well as some other
character from base64 alphabet 4. do something else ;-)
I have tested some popular implementations (such as email clients,
GNU utilities, RTL and other development's libraries). All items
(1)-(4) are actually present.
Is it dangerous? Sure. Consider antiviral software, which implements
behaviour (1), and e-mail client, which implements behaviour
(2). Attacker can insert padding character in the beginning of the
encoded data, and antiviral software will think encoded data is
empty. But e-mail client will think differentother way ;-) So,
bypassing of content-filtering and antiviral protection is obvious
subject for this issue.
How to solve this issue? I believe we should rewrite at least
filtering systems to block malformed base64-encoded data because we
don't know is it malicious or not. Otherwise, we can meet new
powerful e-mail worm.
-----
"Will research information security for food!"
(10727193) /=?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>/(Ombruten)
Kommentar i text 10727322 av Bennett Todd <bet@rahul.net>
Kommentar i text 10732818 av Erwan David <Erwan.David@trusted-logic.fr>
Kommentar i text 10732992 av Birl <sbirl@temple.edu>
Kommentar i text 10733684 av Alexander Ogol <sanyok_nospam@prophysoft.org.ua>
Kommentar i text 10749210 av Earl Hood <earl@earlhood.com>
10727322 2003-09-22 12:59 -0400 /40 rader/ Bennett Todd <bet@rahul.net>
Importerad: 2003-09-22 19:57 av Brevbäraren
Extern mottagare: Ilya Teterin <alienhard@mail.ru>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29134>
Kommentar till text 10727193 av =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
Ärende: Re: base64
------------------------------------------------------------
From: Bennett Todd <bet@rahul.net>
To: Ilya Teterin <alienhard@mail.ru>
Cc: bugtraq@securityfocus.com
Message-ID: <20030922165908.GB25884@rahul.net>
2003-09-22T08:49:59 "Ilya Teterin" :
> Consider we decoding data which contains padding character ('=')
> at the unexpected place. What we should do with such data? The
> specification of base64 decoding does not tell us what we MUST or
> even MAY do with such data... So, we can do anything we like to do:
>
> 1. threat padding character as end of the encoded data
> 2. ignore padding character
> 3. decode padding character as well as some other character from base64 alphabet
> 4. do something else ;-)
>
> I have tested some popular implementations (such as email clients,
> GNU utilities, RTL and other development's libraries). All items
> (1)-(4) are actually present.
Good research!
Another class of variation, producing different handling by
different programs, is weirdly wrapped base64 encoding.
There are many, many ambiguities in email processing. MIME is
positively filthy with them. What happens with various technically
invalid headers varies from processor to processor, charsets are
weird, some email clients do truly bizarre things with filenames in
MIME headers, etc.
A robust email malware scanning system positively has to
canonicalize all this before scanning, to a common and
straightforwad set of usage patterns that are handled the same by
all clients, and by the scanner itself. Some shops do this already;
I don't know of a robust open source component for this yet. But
that may just reflect my ignorance. Probably does:-).
-Bennett
(10727322) /Bennett Todd <bet@rahul.net>/-----------
Bilaga (application/pgp-signature) i text 10727323
10727323 2003-09-22 12:59 -0400 /8 rader/ Bennett Todd <bet@rahul.net>
Importerad: 2003-09-22 19:57 av Brevbäraren
Extern mottagare: Ilya Teterin <alienhard@mail.ru>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29135>
Bilaga (application/pgp-signature) till text 10727322
Ärende: Bilaga till: Re: base64
------------------------------------------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE/bypcHZWg9mCTffwRAkIcAKCHyibcX/fEPrJNmsHhkLdNhaYWjgCfWyGp
UJgo4u1uNCq0wG2mUQiDRHI=
=MF+D
-----END PGP SIGNATURE-----
(10727323) /Bennett Todd <bet@rahul.net>/-----------
10732818 2003-09-23 08:44 +0200 /30 rader/ Erwan David <Erwan.David@trusted-logic.fr>
Importerad: 2003-09-23 19:06 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29138>
Kommentar till text 10727193 av =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
Ärende: Re: base64
------------------------------------------------------------
From: Erwan David <Erwan.David@trusted-logic.fr>
To: bugtraq@securityfocus.com
Message-ID: <20030923064443.GD10162@brehat.trusted-logic.fr>
Le Mon 22/09/2003, "Ilya Teterin" disait
> Consider we decoding data which contains padding character ('=') at the unexpected place. What we should do with such data? The specification of base64 decoding does not tell us what we MUST or even MAY do with such data... So, we can do anything we like to do:
>
> 1. threat padding character as end of the encoded data
> 2. ignore padding character
> 3. decode padding character as well as some other character from
> base64 alphabet
> 4. do something else ;-)
>
> I have tested some popular implementations (such as email clients,
> GNU utilities, RTL and other development's libraries). All items
> (1)-(4) are actually pre
I'll add 5. consider data encoding as broken and return an error. The
base64 specification specifies how to encode. If you get padding
character inside data, it's no more base64 encoded data.
--
Erwan David
==========================================================
Trusted Logic Tel: +33 1 30 97 25 03
5 rue du Bailliage Std: +33 1 30 97 25 00
78000 Versailles Fax: +33 1 30 97 25 19
France
(10732818) /Erwan David <Erwan.David@trusted-logic.fr>/(Ombruten)
10732992 2003-09-23 12:18 -0400 /60 rader/ Birl <sbirl@temple.edu>
Importerad: 2003-09-23 19:57 av Brevbäraren
Extern mottagare: BugTraq@securityfocus.com
Externa svar till: BugTraq@securityfocus.com
Mottagare: Bugtraq (import) <29144>
Kommentar till text 10727193 av =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
Ärende: Re: base64
------------------------------------------------------------
From: Birl <sbirl@temple.edu>
To: BugTraq@securityfocus.com
Message-ID: <Pine.GSO.4.58.0309231211460.16663@concept.ocis.temple.edu>
As it was written on Sep 22, thus "Ilya Teterin" typed:
alienhard: Consider we decoding data which contains padding character
alienhard: ('=') at the unexpected place. What we should do with such
alienhard: data? The specification of base64 decoding does not tell us
alienhard: what we MUST or even MAY do with such data... So, we can do
alienhard: anything we like to do:
alienhard:
alienhard: 1. threat padding character as end of the encoded data
alienhard: 2. ignore padding character
alienhard: 3. decode padding character as well as some other character from base64 alphabet
alienhard: 4. do something else ;-)
alienhard:
alienhard: I have tested some popular implementations (such as email
alienhard: clients, GNU utilities, RTL and other development's
alienhard: libraries). All items (1)-(4) are actually present.
alienhard:
alienhard: Is it dangerous? Sure. Consider antiviral software, which
alienhard: implements behaviour (1), and e-mail client, which implements
alienhard: behaviour (2). Attacker can insert padding character in the
alienhard: beginning of the encoded data, and antiviral software will
alienhard: think encoded data is empty. But e-mail client will think
alienhard: differentother way ;-) So, bypassing of content-filtering and
alienhard: antiviral protection is obvious subject for this issue.
alienhard:
alienhard: How to solve this issue? I believe we should rewrite at least
alienhard: filtering systems to block malformed base64-encoded data
alienhard: because we don't know is it malicious or not. Otherwise, we
alienhard: can meet new powerful e-mail worm.
alienhard:
alienhard: -----
alienhard: "Will research information security for food!"
Excuse my ignorance. I tried to pook around some B64 attachements in
my email files for an answer.
Are you stating that an =
1) should not appear in B64 at all
2) should not appear in the middle of a line, but only at the EOLN
3) should only appear at the end of a B64 file
Answering that question could help me better determine how to write a
procmail filter for this.
Next question: Does the = actually corrupt an attachment, assuming it's a
legimate B64 file, or does the decode still succeed?
Thanks
Birl
(10732992) /Birl <sbirl@temple.edu>/------(Ombruten)
Kommentar i text 10738394 av Lothar Kimmeringer <bugtraq@kimmeringer.de>
10738394 2003-09-23 20:10 +0200 /41 rader/ Lothar Kimmeringer <bugtraq@kimmeringer.de>
Importerad: 2003-09-24 18:36 av Brevbäraren
Extern mottagare: BugTraq@securityfocus.com <BugTraq@securityfocus.com>
Mottagare: Bugtraq (import) <29159>
Kommentar till text 10732992 av Birl <sbirl@temple.edu>
Ärende: Re: base64
------------------------------------------------------------
From: "Lothar Kimmeringer" <bugtraq@kimmeringer.de>
To: "BugTraq@securityfocus.com" <BugTraq@securityfocus.com>
Message-ID: <E1A1rcU-0003rc-00@mrelayng.kundenserver.de>
On Tue, 23 Sep 2003 12:18:31 -0400 (EDT), Birl wrote:
>Excuse my ignorance. I tried to pook around some B64 attachements in my
>email files for an answer.
>
>
>Are you stating that an =
>
>1) should not appear in B64 at all
>2) should not appear in the middle of a line, but only at the EOLN
>3) should only appear at the end of a B64 file
See the corresponding RFC. The number of characters in a base64-coded
text must be a multiply of 4. So ='s are used if there aren't enough
characters and are added at the end of the text.
= is not a valid character inside Base64 and an encoder should stop
with an error or stops decoding.
>Answering that question could help me better determine how to write a
>procmail filter for this.
/.*=[^=]/
(untested)
Regards, Lothar
--
Lothar Kimmeringer E-Mail: mailbody@kimmeringer.de
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)
Always remember: The answer is forty-two, there can only be wrong
questions!
(10738394) /Lothar Kimmeringer <bugtraq@kimmeringer.de>/
Kommentar i text 10739642 av David Wilson <David.Wilson@isode.com>
Kommentar i text 10740269 av Seth Breidbart <sethb@panix.com>
10739642 2003-09-24 19:24 +0100 /78 rader/ David Wilson <David.Wilson@isode.com>
Importerad: 2003-09-24 22:29 av Brevbäraren
Extern mottagare: Lothar Kimmeringer <bugtraq@kimmeringer.de>
Extern kopiemottagare: BugTraq@securityfocus.com <BugTraq@securityfocus.com>
Mottagare: Bugtraq (import) <29179>
Kommentar till text 10738394 av Lothar Kimmeringer <bugtraq@kimmeringer.de>
Ärende: Re: base64
------------------------------------------------------------
From: David Wilson <David.Wilson@isode.com>
To: Lothar Kimmeringer <bugtraq@kimmeringer.de>
Cc: "BugTraq@securityfocus.com" <BugTraq@securityfocus.com>
Message-ID: <1064427873.31228.21.camel@delta.isode.net>
On Tue, 2003-09-23 at 19:10, Lothar Kimmeringer wrote:
> On Tue, 23 Sep 2003 12:18:31 -0400 (EDT), Birl wrote:
>
> >Excuse my ignorance. I tried to pook around some B64 attachements in my
> >email files for an answer.
> >
> >
> >Are you stating that an =
> >
> >1) should not appear in B64 at all
> >2) should not appear in the middle of a line, but only at the EOLN
> >3) should only appear at the end of a B64 file
>
> See the corresponding RFC. The number of characters in a base64-coded
> text must be a multiply of 4. So ='s are used if there aren't enough
> characters and are added at the end of the text.
>
> = is not a valid character inside Base64 and an encoder should stop
> with an error or stops decoding.
>
> >Answering that question could help me better determine how to write a
> >procmail filter for this.
>
> /.*=[^=]/
> (untested)
RFC 2045 states (section 6.8):
"The encoded output stream must be represented in lines of no more
than 76 characters each. All line breaks or other characters not
found in Table 1 must be ignored by decoding software. In base64
data, characters other than those in Table 1, line breaks, and
other white space probably indicate a transmission error, about
which a warning message or even a message rejection might be
appropriate under some circumstances."
So, characters outside the 65 character set are "ignored", but
"warning" or "rejection" might be appropriate. Nicely ambiguous. This
does not apply strictly to '=', but might taken to apply to '='
within a base64 encoding.
"Because it is used only for padding at the end of the data, the
occurrence of any "=" characters may be taken as evidence that the
end of the data has been reached (without truncation in transit).
No such assurance is possible, however, when the number of octets
transmitted was a multiple of three and no "=" characters are
present."
This is also not clear. Does it mean that '=' can be taken to delimit
the data? Or does it mean no more than finding a '=' that means no
data has been lost?
So, there is ambiguity in RFC 2045, and this is the point of the
original post. Different people, and therefore different
implementations will have different interpretations. There is
therefore potential for a vulnerability when checks are performed
using one interpretation but the actual receiver uses another
interpretation.
It would be nice to be able to enforce rules in email servers - there
are many ways in which messages do not conform to the standards - even
when they are unambiguous. But there are too many common email user
agents which generate non-conforming messages.
Or should we reject all these broken messages? ;-)
cheers
David Wilson David.Wilson@isode.com
Isode Limited Tel: +44 (0) 20 8783 2961
http://www.isode.com
(10739642) /David Wilson <David.Wilson@isode.com>/(Ombruten)
Kommentar i text 10743793 av Earl Hood <earl@earlhood.com>
Kommentar i text 10745336 av Christian Vogel <chris@obelix.hedonism.cx>
10743793 2003-09-24 19:27 -0500 /116 rader/ Earl Hood <earl@earlhood.com>
Importerad: 2003-09-25 18:57 av Brevbäraren
Extern mottagare: BugTraq@securityfocus.com <BugTraq@securityfocus.com>
Extern kopiemottagare: David.Wilson@isode.com
Extern kopiemottagare: bugtraq@kimmeringer.de
Externa svar till: earl@earlhood.com
Mottagare: Bugtraq (import) <29219>
Kommentar till text 10739642 av David Wilson <David.Wilson@isode.com>
Ärende: Re: base64
------------------------------------------------------------
From: Earl Hood <earl@earlhood.com>
To: "BugTraq@securityfocus.com" <BugTraq@securityfocus.com>
Cc: David.Wilson@isode.com, bugtraq@kimmeringer.de
Message-ID: <200309250027.h8P0Rrc14860@gator.earlhood.com>
On September 24, 2003 at 19:24, David Wilson wrote:
> RFC 2045 states (section 6.8):
>
> "The encoded output stream must be represented in lines of no more
> than 76 characters each. All line breaks or other characters not
> found in Table 1 must be ignored by decoding software. In base64
> data, characters other than those in Table 1, line breaks, and other
> white space probably indicate a transmission error, about which a
> warning message or even a message rejection might be appropriate
> under some circumstances."
>
> So, characters outside the 65 character set are "ignored", but "warning"
> or "rejection" might be appropriate. Nicely ambiguous.
Nowadays it is unclear what the proper action is, but you must
remember that the RFC is dated 1996 (and the first MIME RFC is dated
1992), a time when email security was not as important as it is today
and there was probably a deference to not put too many "must"s on
implementors or to let user's have the opportunity to "recover" badly
transmitted data.
You'd need to contact the RFC authors to determine what they were
thinking when the warning or rejection text as only advisory vs
mandatory.
> This does not
> apply strictly to '=', but might taken to apply to '=' within a base64
> encoding.
>
> "Because it is used only for padding at the end of the data, the
> occurrence of any "=" characters may be taken as evidence that the
> end of the data has been reached (without truncation in transit). No
> such assurance is possible, however, when the number of octets
> transmitted was a multiple of three and no "=" characters are
> present."
>
> This is also not clear. Does it mean that '=' can be taken to delimit
> the data? Or does it mean no more than finding a '=' that means no data
> has been lost?
The '=' is not always present, but when it is, it occurs at the end
of data. The statement you quote about a '=' not being present
cannot, by itself, be used to assume that the data has been truncated
since the existence of '=' is optional. I.e. Other mechanisms
outside of base64 encoding itself must be used to determine if
truncation has occured.
> So, there is ambiguity in RFC 2045, and this is the point of the
> original post. Different people, and therefore different implementations
> will have different interpretations. There is therefore potential for a
> vulnerability when checks are performed using one interpretation but the
> actual receiver uses another interpretation.
This will only happen if the server check is forgiving about "errors"
in the base64 data vs only allowing proper "well-formed" base64 data
through.
> It would be nice to be able to enforce rules in email servers - there
> are many ways in which messages do not conform to the standards - even
> when they are unambiguous. But there are too many common email user
> agents which generate non-conforming messages.
I think you are over generalizing to other parts of email vs the
issue of base64 data. Can you name one MUA that generates improper
base64 entities? When just addressing the base64 encoding/decoding
issue, a reasonable set of rules can be established for dealing with
it since the permutations of variations is small.
Each other mail formatting problem will have to be dealt with
separately. I do advocate that the MIME RFC maintainers be contacted
to make things more clear (hence the term "Request For Comments")
or at least get their thoughts on the matter, but as we all know,
even when a standard is clear about something, there will be software
that is non-conforming and that security folks will have to deal with.
You may also get the case where it is reasonable in some situations
where reasonable people will disagree on what do about something.
For example, should bad characters be in base64 data cause a
rejection, or should some form of error recovery be allowed? Who is
to say one is always right and the other is always wrong? It may be
best to state what best practices are for these cases vs mandating a
specific action.
> Or should we reject all these broken messages? ;-)
It seems the following can be applied by servers and other software
trying to detect for malicious data:
* If there are any non-base64 characters and security is to be as
strict as possible, reject the message, or at a minimum the entity
that is base64 encoded.
* If there is any non-whitespace data after an '=', or '==' sequence,
reject the entity if strict security is required since it is just
a variation of item 1.
Basically, assume that MUAs are going to be as lenient as possible.
Viral detection software should alway imply the strictest decoding
rules, and for properly encoded data, this is a safe bet. I
personally have never seen a case of base64 encoded entities that
included non-base64 characters or '='s in the middle of encoded data.
Therefore, if such a case does happen, it is either a transmission
problem or something malicious being attempted.
--ewh
--
Earl Hood, <earl@earlhood.com>
Web: <http://www.earlhood.com/>
PGP Public Key: <http://www.earlhood.com/gpgpubkey.txt>
(10743793) /Earl Hood <earl@earlhood.com>/(Ombruten)
10745336 2003-09-25 09:10 +0200 /49 rader/ Christian Vogel <chris@obelix.hedonism.cx>
Importerad: 2003-09-26 03:42 av Brevbäraren
Extern mottagare: David Wilson <David.Wilson@isode.com>
Extern kopiemottagare: Lothar Kimmeringer <bugtraq@kimmeringer.de>
Extern kopiemottagare: BugTraq@securityfocus.com <BugTraq@securityfocus.com>
Mottagare: Bugtraq (import) <29250>
Kommentar till text 10739642 av David Wilson <David.Wilson@isode.com>
Ärende: Re: base64
------------------------------------------------------------
From: Christian Vogel <chris@obelix.hedonism.cx>
To: David Wilson <David.Wilson@isode.com>
Cc: Lothar Kimmeringer <bugtraq@kimmeringer.de>,
"BugTraq@securityfocus.com" <BugTraq@securityfocus.com>
Message-ID: <20030925091004.A8591@obelix.frop.org>
Hi David,
> RFC 2045 states (section 6.8):
> data, characters other than those in Table 1, line breaks, and other
> white space probably indicate a transmission error, about which a
> warning message or even a message rejection might be appropriate
> under some circumstances."
A user-agent has to assume that it's message might be dropped if it
creates base64 with junk in it. So it should not create these things
and it's perfectly resaonable for a MTA/virus-scanner to drop those
messages.
> "Because it is used only for padding at the end of the data, the
> occurrence of any "=" characters may be taken as evidence that the
> end of the data has been reached (without truncation in transit). No
> such assurance is possible, however, when the number of octets
> transmitted was a multiple of three and no "=" characters are
> present."
Again, as the mail-client does not have a way to know how the
generated data is interpreted in those ambigous cases its reasonable
to just drop those messages.
> But there are too many common email user
> agents which generate non-conforming messages.
Is there already a list of broken MUAs? Do the vendors even know
(yes, they should have cought that during testing... ;-) )
> Or should we reject all these broken messages? ;-)
Either reject them or convert them to a canonical form. But that will
generate further problems, e.g. if you modify signed payload that way.
Chris
--
Message passing as the fundamental operation of the OS is just an
excercise in computer science masturbation. It may feel good, but you
don't actually get anything DONE. -- Linus Torvalds
(10745336) /Christian Vogel <chris@obelix.hedonism.cx>/(Ombruten)
10740269 2003-09-24 13:01 -0400 /24 rader/ Seth Breidbart <sethb@panix.com>
Importerad: 2003-09-25 01:03 av Brevbäraren
Extern mottagare: BugTraq@securityfocus.com
Mottagare: Bugtraq (import) <29195>
Kommentar till text 10738394 av Lothar Kimmeringer <bugtraq@kimmeringer.de>
Ärende: Re: base64
------------------------------------------------------------
From: Seth Breidbart <sethb@panix.com>
To: BugTraq@securityfocus.com
Message-ID: <200309241701.h8OH1Lt15303@panix5.panix.com>
> See the corresponding RFC. The number of characters in a base64-coded
> text must be a multiply of 4. So ='s are used if there aren't enough
> characters and are added at the end of the text.
>
> = is not a valid character inside Base64 and an encoder should stop
> with an error or stops decoding.
That depends on the purpose of the decoder.
If your anti-virus decoder stops there, but a virus writer knows that
the decoder used by some popular mail client keeps going, then your
program isn't going to detect his virus.
If you're looking for something that might be hidden, then you need to
decode in _every_ way that _any_ mail client (that you support) does,
lest you have false negatives.
Seth
(10740269) /Seth Breidbart <sethb@panix.com>/-------
10733684 2003-09-23 19:50 +0300 /30 rader/ Alexander Ogol <sanyok_nospam@prophysoft.org.ua>
Importerad: 2003-09-23 22:15 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29157>
Kommentar till text 10727193 av =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
Ärende: Re: base64
------------------------------------------------------------
From: Alexander Ogol <sanyok_nospam@prophysoft.org.ua>
To: bugtraq@securityfocus.com
Message-ID: <1082234495@prophysoft.org.ua>
>>>>> "IT" == Ilya Teterin writes:
[kham-kham-kham....argh....]
IT> How to solve this issue? I believe we should rewrite at least
IT> filtering systems to block malformed base64-encoded data because
we IT> don't know is it malicious or not. Otherwise, we can meet new
powerful IT> e-mail worm.
Rewriting filtering systems for rejecting malformed mail couldn't be
good decision in all situations. Some mailing lists (debian-russian,
for example) add some 7bit information after letter body while
re-forwarding, regardless of was the letter base64/QP encoded or not,
resulting of such malformed mail (and if your mail server recodes all
8bit mail to base64, you have no way to write to those mail lists
without creating malformed mail at result). So I think that the
right solution (before antivirus software would be rewritten) is to
write filters by yourself - decode base64 as that do popular mail
clients and give them to antivirus. Of course, it's somehow more
hard than just filtering up all those letters.
--
Sincerely yours, Alexander Ogol, happy GNU/Linux user.
(10733684) /Alexander Ogol <sanyok_nospam@prophysoft.org.ua>/(Ombruten)
Kommentar i text 10738768 av Christian Vogel <chris@obelix.hedonism.cx>
10738768 2003-09-24 09:09 +0200 /40 rader/ Christian Vogel <chris@obelix.hedonism.cx>
Importerad: 2003-09-24 20:06 av Brevbäraren
Extern mottagare: Alexander Ogol <sanyok_nospam@prophysoft.org.ua>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29173>
Kommentar till text 10733684 av Alexander Ogol <sanyok_nospam@prophysoft.org.ua>
Ärende: Re: base64
------------------------------------------------------------
From: Christian Vogel <chris@obelix.hedonism.cx>
To: Alexander Ogol <sanyok_nospam@prophysoft.org.ua>
Cc: bugtraq@securityfocus.com
Message-ID: <20030924090943.A6330@obelix.frop.org>
Hi,
On Tue, Sep 23, 2003 at 07:50:56PM +0300, Alexander Ogol wrote:
> decision in all situations. Some mailing lists (debian-russian, for example)
> add some 7bit information after letter body while re-forwarding, regardless
> of was the letter base64/QP encoded or not, resulting of such malformed
Then this software is severly broken (MIME-wise), imho, and needs to
be updated/changed/dumed.
> So I think that the right solution (before antivirus software would be
> rewritten) is to write filters by yourself - decode base64 as that do
> popular mail clients and give them to antivirus.
With this approach, you are always on the "one step behind" side of
the problem. It's only a matter of time that someone finds out that
(made up example:) you can use a UTF8-mis-encoded "=" in Microsoft's
base64-decoder... The only sane way is to check if it's in the
standard-form ("abcABC=") and reject or convert if it's not.
99.99% of all software should create the standard form, so please
let the tiny fraction of users with broken software suffer
when their mails get rejected.
(Note: this of course applies not only to Base64 but to all aspects
of header-parsing, file-format guessing etc...)
Chris
--
01234567 <- The amazing* indent-o-meter!
^ (*: Indent-o-meter may not actually amaze.)
-- stolen from Nick Moffitt nick(at)zork.net
(10738768) /Christian Vogel <chris@obelix.hedonism.cx>/(Ombruten)
Kommentar i text 10739456 av der Mouse <mouse@Rodents.Montreal.QC.CA>
Kommentar i text 10739908 av David Wilson <David.Wilson@isode.com>
10739456 2003-09-24 14:30 -0400 /36 rader/ der Mouse <mouse@Rodents.Montreal.QC.CA>
Importerad: 2003-09-24 22:05 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29178>
Kommentar till text 10738768 av Christian Vogel <chris@obelix.hedonism.cx>
Ärende: Re: base64
------------------------------------------------------------
From: der Mouse <mouse@Rodents.Montreal.QC.CA>
To: bugtraq@securityfocus.com
Message-ID: <200309241841.OAA16656@Sparkle.Rodents.Montreal.QC.CA>
> 99.99% of all software should create the standard form, so please let
> the tiny fraction of users with broken software suffer when their
> mails get rejected.
> (Note: this of course applies not only to Base64 but to all aspects
> of header-parsing, file-format guessing etc...)
I agree with you that such errors should produce rejections.
But there's an awful lot of otherwise legitimate mail software out
there that commits (and more that accepts and passes along)
rudimentary errors like
- One-digit hour, minute, or second values
- Invalid timezone strings ("Pacific Standard Time", "CEST", "JST",
etc)
- Completely missing timezones
- Non-encoded high-half characters in header text
(From: "François" <fxm@example.net>)
- High-half characters in mail implicitly marked charset=US-ASCII
- Non-8859 characters (0x80-0x9f) in mail marked charset=iso-8859-1
- Retrying messages that have been 5xx-rejected
Indeed, I recently discovered I had been auto-unsubbed from this very
list for, as near as I can tell, rejecting mail exhibiting some of
those syntax errors.
/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML mouse@rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
(10739456) /der Mouse <mouse@Rodents.Montreal.QC.CA>/(Ombruten)
10739908 2003-09-24 20:01 +0100 /52 rader/ David Wilson <David.Wilson@isode.com>
Importerad: 2003-09-24 23:14 av Brevbäraren
Extern mottagare: Christian Vogel <chris@obelix.hedonism.cx>
Extern kopiemottagare: Alexander Ogol <sanyok_nospam@prophysoft.org.ua>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29181>
Kommentar till text 10738768 av Christian Vogel <chris@obelix.hedonism.cx>
Ärende: Re: base64
------------------------------------------------------------
From: David Wilson <David.Wilson@isode.com>
To: Christian Vogel <chris@obelix.hedonism.cx>
Cc: Alexander Ogol <sanyok_nospam@prophysoft.org.ua>,
bugtraq@securityfocus.com
Message-ID: <1064430111.31234.41.camel@delta.isode.net>
>
> 99.99% of all software should create the standard form, so please
> let the tiny fraction of users with broken software suffer
> when their mails get rejected.
>
> (Note: this of course applies not only to Base64 but to all aspects
> of header-parsing, file-format guessing etc...)
I wish it were true...
I see many invalid messages, e.g.
- Using CR or LF not as CR LF, or including NUL. (Forbidden for MIME
messages, except for Content-transfer-encoding: binary, which needs an
SMTP Extension which few MTAs support).
- Using encoded words in MIME header parameter values (forbidden by
RFC 2047 Page 8).
- Not quoting parameters when required
- Not setting Content-transfer-encoding: 8bit on all enclosing MIME
composite types (multipart or message/rfc822) when 8bit is used in a
data part (see RFC 2045 Section 6.4).
- Using characters outside the advertised character set. (E.g. using
the MS 'smart quotes' in something advertised as iso-8859-1).
- Not using the 'minimum' character set (e.g. saying iso-8859-1 when
us-ascii would do).
- Using 8bit characters in headers.
That's just in the structure of the message.
The original post in this thread made the valid point that different
interpreters of MIME messages make different assumptions. That applies
to other areas as well as base64 decoding.
cheers
David Wilson David.Wilson@isode.com
Isode Limited Tel: +44 (0) 20 8783 2961
http://www.isode.com
(10739908) /David Wilson <David.Wilson@isode.com>/(Ombruten)
10749210 2003-09-26 15:38 -0500 /34 rader/ Earl Hood <earl@earlhood.com>
Importerad: 2003-09-26 23:30 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Externa svar till: earl@earlhood.com
Mottagare: Bugtraq (import) <29296>
Kommentar till text 10727193 av =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
Ärende: Re: base64
------------------------------------------------------------
From: Earl Hood <earl@earlhood.com>
To: bugtraq@securityfocus.com
Message-ID: <200309262038.h8QKcNI31094@gator.earlhood.com>
Those following this thread may be interested in checking out RFC
3548: <http://www.faqs.org/rfcs/rfc3548.html>. It appears to address
some of the concerns raised in this thread and what the recommended
practices are.
It is worth noting the RFC 2045 states that decoders *must* ignore
invalid characters when decoding. However, I believe RFC 2045 is
ambiguous about '='. If it appears in the middle of input stream,
it could imply end-of-data or an invalid character if you take the
approach that '=' is only a valid base64 character at the end of
encoded data.
The following is written in the Security Considerations of
RFC 3548:
When implementing Base encoding and decoding, care should be taken
not to introduce vulnerabilities to buffer overflow attacks, or
other attacks on the implementation. A decoder should not break
on invalid input including, e.g., embedded NUL characters (ASCII
0).
If non-alphabet characters are ignored, instead of causing
rejection of the entire encoding (as recommended), a covert
channel that can be used to "leak" information is made possible.
The implications of this should be understood in applications that
do not follow the recommended practice.
--ewh
(10749210) /Earl Hood <earl@earlhood.com>/(Ombruten)
10732949 2003-09-22 17:52 -0700 /69 rader/ <latte@hushmail.com>
Importerad: 2003-09-23 19:42 av Brevbäraren
Extern mottagare: alienhard@mail.ru
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29143>
Ärende: RE: base64
------------------------------------------------------------
From: <latte@hushmail.com>
To: alienhard@mail.ru, bugtraq@securityfocus.com
Cc:
Message-ID: <200309230052.h8N0qQLK043440@mailserver2.hushmail.com>
What about this section:
" Because it is used only for padding at the end of the data, the
occurrence of any '=' characters may be taken as evidence that the
end of the data has been reached (without truncation in transit). No
such assurance is possible, however, when the number of octets
transmitted was a multiple of three. "
From: http://www.ietf.org/rfc/rfc1521.txt?number=1521
Seems to suggest that it should be treated as the end of
input.
-----Original Message-----
From: "Ilya Teterin" [mailto:alienhard@mail.ru]
Sent: Monday, 22 September 2003 10:50 PM
To: bugtraq@securityfocus.com
Subject: base64
Consider we decoding data which contains padding character ('=') at
the unexpected place. What we should do with such data? The
specification of base64 decoding does not tell us what we MUST or
even MAY do with such data... So, we can do anything we like to do:
1. threat padding character as end of the encoded data 2. ignore
padding character 3. decode padding character as well as some other
character from base64 alphabet 4. do something else ;-)
I have tested some popular implementations (such as email clients, GNU
utilities, RTL and other development's libraries). All items (1)-(4)
are actually present.
Is it dangerous? Sure. Consider antiviral software, which implements
behaviour (1), and e-mail client, which implements behaviour
(2). Attacker can insert padding character in the beginning of the
encoded data, and antiviral software will think encoded data is
empty. But e-mail client will think differentother way ;-) So,
bypassing of content-filtering and antiviral protection is obvious
subject for this issue.
How to solve this issue? I believe we should rewrite at least
filtering systems to block malformed base64-encoded data because we
don't know is it malicious or not. Otherwise, we can meet new
powerful e-mail worm.
-----
"Will research information security for food!"
Concerned about your privacy? Follow this link to get
FREE encrypted email: https://www.hushmail.com/?l=2
Free, ultra-private instant messaging with Hush Messenger
https://www.hushmail.com/services.php?subloc=messenger&l=434
Promote security and make money with the Hushmail Affiliate Program:
https://www.hushmail.com/about.php?subloc=affiliate&l=427
(10732949) /<latte@hushmail.com>/---------(Ombruten)
10733487 2003-09-23 21:38 +0400 /16 rader/ =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
Importerad: 2003-09-23 21:51 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Externa svar till: alienhard@mail.ru
Mottagare: Bugtraq (import) <29156>
Ärende: Re: base64
------------------------------------------------------------
From: =?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>
To: bugtraq@securityfocus.com
Message-ID: <E1A1r72-000Len-00.alienhard-mail-ru@f21.mail.ru>
Erwan David:
> I'll add 5. consider data encoding as broken and return an error.
> The base64 specification specifies how to encode. If you get
> padding character inside data, it's no more base64 encoded data.
The base64 specification specifies how to decode, too. And it
specifies how to handle unexpected characters during
decoding. Someone... may assume that specification of decoding is
complete :-)
Does anyone know product which actually implements (5)?
Most beautiful product was MS Outlook Express. When it meet padding
in the first position, it will decode it as "A". In other positions
padding will be decoded as "A" too, but some characters from decoded
text will be deleted using "pattern" depending on position of
padding. I doubt there is content filtering program which implements
the SAME way of base64 decoding.
(10733487) /=?koi8-r?Q?=22?=Ilya Teterin=?koi8-r?Q?=22=20?= <alienhard@mail.ru>/(Ombruten)
10739962 2003-09-24 15:13 -0400 /64 rader/ MightyE <trash@mightye.org>
Importerad: 2003-09-24 23:27 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29182>
Ärende: Re: base64
------------------------------------------------------------
From: MightyE <trash@mightye.org>
To: bugtraq@securityfocus.com
Message-ID: <3F71ECF5.4050406@mightye.org>
I agree, I don't think it's unreasonable to reject improperly
formatted messages. Chances are much higher that they're spam or
virii, and the minority with broken clients will find their way to
non-broken clients. If you are parsing the message, particularly
looking for malware, and encounter an improper encoding, bounce the
message with a meaningful error, this way you don't have to worry
about a targeted exploit that depends on the way one email client
(mis)interprets a message in a different way from your virus scanner.
The RFC does declare an = to indicate that the end of the data stream
has been reached, further data should be truncated, though it seems
each email client actually handles this differently. Take the low
road catchall, and simply reject them as a matter of course.
-Eric Stevens
mightye a@t mightye d.o.t org
Christian Vogel wrote:
>Hi,
>
>On Tue, Sep 23, 2003 at 07:50:56PM +0300, Alexander Ogol wrote:
>
>
>>decision in all situations. Some mailing lists (debian-russian, for example)
>>add some 7bit information after letter body while re-forwarding, regardless
>>of was the letter base64/QP encoded or not, resulting of such malformed
>>
>>
>
>Then this software is severly broken (MIME-wise), imho, and needs to be
>updated/changed/dumed.
>
>
>
>>So I think that the right solution (before antivirus software would be
>>rewritten) is to write filters by yourself - decode base64 as that do
>>popular mail clients and give them to antivirus.
>>
>>
>
>With this approach, you are always on the "one step behind" side of
>the problem. It's only a matter of time that someone finds out that
>(made up example:) you can use a UTF8-mis-encoded "=" in Microsoft's
>base64-decoder... The only sane way is to check if it's in the
>standard-form ("abcABC=") and reject or convert if it's not.
>
>99.99% of all software should create the standard form, so please
>let the tiny fraction of users with broken software suffer
>when their mails get rejected.
>
>(Note: this of course applies not only to Base64 but to all aspects
> of header-parsing, file-format guessing etc...)
>
> Chris
>
>
>
(10739962) /MightyE <trash@mightye.org>/--(Ombruten)
Kommentar i text 10740414 av Buck Huppmann <buckh@pobox.com>
10740414 2003-09-24 16:49 -0400 /27 rader/ Buck Huppmann <buckh@pobox.com>
Importerad: 2003-09-25 03:09 av Brevbäraren
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29203>
Kommentar till text 10739962 av MightyE <trash@mightye.org>
Ärende: Re: base64
------------------------------------------------------------
From: Buck Huppmann <buckh@pobox.com>
To: MightyE: ;
Cc: bugtraq@securityfocus.com
Message-ID: <20030924204925.GC614@dsl092-163-146.wdc2.dsl.speakeasy.net>
On Wed, Sep 24, 2003 at 03:13:57PM -0400, MightyE wrote:
> I agree, I don't think it's unreasonable to reject improperly formatted
> messages.
> Take the low road
> catchall, and simply reject them as a matter of course.
this runs counter to the maxim of Postel
<http://www.postel.org/postel.html>:
"Be liberal in what you accept, and conservative in what you send."
-- jon
RFC-1122 (originates in RFC760)
or was that wisdom for a different time?
`"What would Jon have done?", we will think, as we wrestle in the
days ahead with the problems Jon kept so well tamed for so many
years.'
--Vint Cerf
RFC 2468
(10740414) /Buck Huppmann <buckh@pobox.com>/(Ombruten)
Kommentar i text 10743734 av Andrew Church <achurch@achurch.org>
10743734 2003-09-25 14:20 -0000 /32 rader/ Andrew Church <achurch@achurch.org>
Importerad: 2003-09-25 18:45 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29217>
Kommentar till text 10740414 av Buck Huppmann <buckh@pobox.com>
Ärende: Re: base64
------------------------------------------------------------
From: achurch@achurch.org (Andrew Church)
To: bugtraq@securityfocus.com
Message-ID: <3f727ccf.64320@achurch.org>
>On Wed, Sep 24, 2003 at 03:13:57PM -0400, MightyE wrote:
>> I agree, I don't think it's unreasonable to reject improperly formatted
>> messages.
>
>> Take the low road
>> catchall, and simply reject them as a matter of course.
>
>this runs counter to the maxim of Postel
><http://www.postel.org/postel.html>:
>
>"Be liberal in what you accept, and conservative in what you send."
It depends how you interpret it. I'd interpret the rejection as
being conservative on the sending side--leaving the data as is would
violate _that_. I could see an argument for correcting malformed
data where possible (e.g. cutting off all data after the first =
sign), but that raises the question of how to know that you're
transmitting the same content that the sender intended. Particularly
in cases like this, where the standard is ambiguous, the only "safe"
options with respect to preserving the content are to send the data
on as is or reject it entirely, and in that case I think Postel's
maxim would lean toward rejection; the "liberal" part is only for
data you can accurately interpret.
--Andrew Church
achurch@achurch.org
http://achurch.org/
(10743734) /Andrew Church <achurch@achurch.org>/(Ombruten)
10743723 2003-09-25 09:06 -0400 /153 rader/ MightyE <trash@mightye.org>
Importerad: 2003-09-25 18:43 av Brevbäraren
Extern mottagare: Lawrence MacIntyre <lpz@ornl.gov>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29216>
Ärende: Re: base64
------------------------------------------------------------
From: MightyE <trash@mightye.org>
To: Lawrence MacIntyre <lpz@ornl.gov>
Cc: bugtraq@securityfocus.com
Message-ID: <3F72E872.7000007@mightye.org>
Lawrence MacIntyre wrote:
>Whatever happened to the strategy:
>
>Be conservative in what you send and liberal in what you accept.
>
>
>
This strategy falls on its face when you're looking at scanning emails
for viruses. If you're liberal in what you accept, then someone has the
ability to encode their virus, targeted toward a specific mail program,
which behaves in a certain way when it encounters improperly encoded
base64 data.
There are two methods which you can use in the writing of your email
virus scanner; you can either decode it every known way that any
client does so, in which case you are only ever able to perform
damage control, and must update your software each time a program
exhibits new behavior. This method also introduces a high load on
the mail server since you have to decode the same attachment many
times, and scan it for viruses many times. If there are only 5
conceivable means to decode an error in the base64 encoding, then
you've introduced 5 times the load on your mail server compared to
traditional approaches. This thus also opens you wider to DoS
attacks.
Alternatively you can accept it only if it is properly encoded,
relying on the 99.9% of clients out there that know how to read an
RFC. If all mail servers had always only accepted proper base64
encoding, then the authors of all email clients would have their
clients encoding correctly from the very start, and you'd never need
to worry about this now (or else they'd never actually get any users
since they'd have a hard time sending attachments).
>The real problem isn't badly formed MIME attachments or blown BASE64
>encoding, it is instead mail clients that execute email as if it were
>code, when we all know that email is simply data. You don't execute
>data...
>
>
>
No, the problem at hand is how best to scan for viruses in incoming
email, that's what the discussion is centered around. This latest
virus going around uses social engineering to get the user to run the
executable; the mail program never runs it. In your paragraph above,
you thus describe this as not a problem, or failing that, this certainly
doesn't fall within your description of what a problem is. I disagree,
and insist that in an increasingly hostile Internet environment,
stricter acceptance policies are needed more and more. Non compliance
with RFC's should not be tolerated, this sort of non compliance is
specifically the sort of loop hole that crackers can exploit to bypass
the likes of virus scanners.
>p.s. If your theory on broken clients were true, then Outlook would be
>gone by now:-)
>
>
>
No, because my theory depends on being conservative in what you accept;
being liberal in what you accept does not give users of broken clients
any incentive to find a functioning client. When users are hit by an
Outlook worm, everything blows over in a week or two, it's a big
inconvenience, but they're again sending emails again, and the problem
is "fixed" as far as they're concerned, so no reason to change now, they
after all have their address book set up in this client. If a worm came
around that was hitting them, and there never *was* a fix for their
client, the user is faced with either dropping off the face of the 'net
for eternity, or finding a new client. I think we're both aware of what
the obvious choice is for the user here.
Yours,
Eric Stevens
mightye a@t mightye d.o.t org
>On Wed, 2003-09-24 at 15:13, MightyE wrote:
>
>
>>I agree, I don't think it's unreasonable to reject improperly formatted
>>messages. Chances are much higher that they're spam or virii, and the
>>minority with broken clients will find their way to non-broken clients.
>>If you are parsing the message, particularly looking for malware, and
>>encounter an improper encoding, bounce the message with a meaningful
>>error, this way you don't have to worry about a targeted exploit that
>>depends on the way one email client (mis)interprets a message in a
>>different way from your virus scanner.
>>
>>The RFC does declare an = to indicate that the end of the data stream
>>has been reached, further data should be truncated, though it seems each
>>email client actually handles this differently. Take the low road
>>catchall, and simply reject them as a matter of course.
>>
>>-Eric Stevens
>>mightye a@t mightye d.o.t org
>>
>>Christian Vogel wrote:
>>
>>
>>
>>>Hi,
>>>
>>>On Tue, Sep 23, 2003 at 07:50:56PM +0300, Alexander Ogol wrote:
>>>
>>>
>>>
>>>
>>>>decision in all situations. Some mailing lists (debian-russian, for example)
>>>>add some 7bit information after letter body while re-forwarding, regardless
>>>>of was the letter base64/QP encoded or not, resulting of such malformed
>>>>
>>>>
>>>>
>>>>
>>>Then this software is severly broken (MIME-wise), imho, and needs to be
>>>updated/changed/dumed.
>>>
>>>
>>>
>>>
>>>
>>>>So I think that the right solution (before antivirus software would be
>>>>rewritten) is to write filters by yourself - decode base64 as that do
>>>>popular mail clients and give them to antivirus.
>>>>
>>>>
>>>>
>>>>
>>>With this approach, you are always on the "one step behind" side of
>>>the problem. It's only a matter of time that someone finds out that
>>>(made up example:) you can use a UTF8-mis-encoded "=" in Microsoft's
>>>base64-decoder... The only sane way is to check if it's in the
>>>standard-form ("abcABC=") and reject or convert if it's not.
>>>
>>>99.99% of all software should create the standard form, so please
>>>let the tiny fraction of users with broken software suffer
>>>when their mails get rejected.
>>>
>>>(Note: this of course applies not only to Base64 but to all aspects
>>>of header-parsing, file-format guessing etc...)
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>>
(10743723) /MightyE <trash@mightye.org>/--(Ombruten)
Kommentar i text 10743801 av Bennett Todd <bet@rahul.net>
10743801 2003-09-25 11:30 -0400 /27 rader/ Bennett Todd <bet@rahul.net>
Importerad: 2003-09-25 18:58 av Brevbäraren
Extern mottagare: MightyE <trash@mightye.org>
Extern kopiemottagare: Lawrence MacIntyre <lpz@ornl.gov>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29220>
Kommentar till text 10743723 av MightyE <trash@mightye.org>
Ärende: Re: base64
------------------------------------------------------------
From: Bennett Todd <bet@rahul.net>
To: MightyE <trash@mightye.org>
Cc: Lawrence MacIntyre <lpz@ornl.gov>, bugtraq@securityfocus.com
Message-ID: <20030925153009.GA5716@rahul.net>
2003-09-25T09:06:58 MightyE:
> There are two methods which you can use in the writing of your
> email virus scanner; you can either decode it every known way that
> any client does so, [...] Alternatively you can accept it only if
> it is properly encoded, [...]
There's a third method, which I think is rather better than either
of those.
You can re-code everything into a canonical form. Some email client
drop some punctuation characters in filenames? Delete all such
characters from filenames. Different tools handle various i18n
encoded filenames differently? Map to US-ASCII. Enforce length
limits. Recode base64. Recode uuencoded chunks. Regularize
non-standard MIME.
Do all this canonicalization before the message hits your
attachment type policy enforcement and malware scanner, so they only
have to deal with the common forms that everybody handles the same.
-Bennett
(10743801) /Bennett Todd <bet@rahul.net>/-----------
Bilaga (application/pgp-signature) i text 10743802
Kommentar i text 10744737 av MightyE <trash@mightye.org>
Kommentar i text 10748424 av Earl Hood <earl@earlhood.com>
Kommentar i text 10748494 av Alun Jones <alun@texis.com>
10743802 2003-09-25 11:30 -0400 /8 rader/ Bennett Todd <bet@rahul.net>
Importerad: 2003-09-25 18:58 av Brevbäraren
Extern mottagare: MightyE <trash@mightye.org>
Extern kopiemottagare: Lawrence MacIntyre <lpz@ornl.gov>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29221>
Bilaga (application/pgp-signature) till text 10743801
Ärende: Bilaga till: Re: base64
------------------------------------------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE/cwoBHZWg9mCTffwRAvX5AJsE0apz9dGZR4r0t8dI2nQM8cTyKQCgjPHC
0Qr/jSKwr6iC/2DaGXoDpao=
=g2Op
-----END PGP SIGNATURE-----
(10743802) /Bennett Todd <bet@rahul.net>/-----------
10744737 2003-09-25 14:21 -0400 /47 rader/ MightyE <trash@mightye.org>
Importerad: 2003-09-25 22:32 av Brevbäraren
Extern mottagare: Bennett Todd <bet@rahul.net>
Extern kopiemottagare: Lawrence MacIntyre <lpz@ornl.gov>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29234>
Kommentar till text 10743801 av Bennett Todd <bet@rahul.net>
Ärende: Re: base64
------------------------------------------------------------
From: MightyE <trash@mightye.org>
To: Bennett Todd <bet@rahul.net>
Cc: Lawrence MacIntyre <lpz@ornl.gov>, bugtraq@securityfocus.com
Message-ID: <3F733235.6010206@mightye.org>
This is a remarkably good and simple approach (thus I'm a bit abashed
that I didn't think of it), and permits liberal acceptance, with out
exposing the user to a targeted intentional malformation of base64
encoded data. Accept all mis formed data, make a best guest as to
what it is (or follow RFC on how to handle this where applicable),
check the result, and pending acceptance, write it to the user's
mail file, in its newly interpreted form.
Eric Stevens
mightye a@t mightye d.o.t org
Bennett Todd wrote:
>2003-09-25T09:06:58 MightyE:
>
>
>>There are two methods which you can use in the writing of your
>>email virus scanner; you can either decode it every known way that
>>any client does so, [...] Alternatively you can accept it only if
>>it is properly encoded, [...]
>>
>>
>
>There's a third method, which I think is rather better than either
>of those.
>
>You can re-code everything into a canonical form. Some email client
>drop some punctuation characters in filenames? Delete all such
>characters from filenames. Different tools handle various i18n
>encoded filenames differently? Map to US-ASCII. Enforce length
>limits. Recode base64. Recode uuencoded chunks. Regularize
>non-standard MIME.
>
>Do all this canonicalization before the message hits your
>attachment type policy enforcement and malware scanner, so they only
>have to deal with the common forms that everybody handles the same.
>
>-Bennett
>
>
(10744737) /MightyE <trash@mightye.org>/--(Ombruten)
10748424 2003-09-25 18:46 -0500 /24 rader/ Earl Hood <earl@earlhood.com>
Importerad: 2003-09-26 19:57 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Extern kopiemottagare: Bennett Todd <bet@rahul.net>
Extern kopiemottagare: MightyE <trash@mightye.org>
Extern kopiemottagare: Lawrence MacIntyre <lpz@ornl.gov>
Externa svar till: earl@earlhood.com
Mottagare: Bugtraq (import) <29269>
Kommentar till text 10743801 av Bennett Todd <bet@rahul.net>
Ärende: Re: base64
------------------------------------------------------------
From: Earl Hood <earl@earlhood.com>
To: bugtraq@securityfocus.com
Cc: MightyE <trash@mightye.org>, Lawrence MacIntyre <lpz@ornl.gov>
Message-ID: <200309252346.h8PNkbU23630@gator.earlhood.com>
On September 25, 2003 at 11:30, Bennett Todd wrote:
> There's a third method, which I think is rather better than either
> of those.
>
> You can re-code everything into a canonical form. Some email client
> drop some punctuation characters in filenames? Delete all such
> characters from filenames. Different tools handle various i18n
> encoded filenames differently? Map to US-ASCII. Enforce length
> limits. Recode base64. Recode uuencoded chunks. Regularize
> non-standard MIME.
You cannot do this for signed messages, therefore, you still
need to either decode in all possible ways or drop the message
(or the offending entity).
--ewh
(10748424) /Earl Hood <earl@earlhood.com>/----------
Kommentar i text 10748375 av Bennett Todd <bet@rahul.net>
Kommentar i text 10749047 av 3APA3A <3APA3A@SECURITY.NNOV.RU>
10748375 2003-09-26 13:08 -0400 /46 rader/ Bennett Todd <bet@rahul.net>
Importerad: 2003-09-26 19:50 av Brevbäraren
Extern mottagare: Earl Hood <earl@earlhood.com>
Extern kopiemottagare: bugtraq@securityfocus.com
Extern kopiemottagare: MightyE <trash@mightye.org>
Extern kopiemottagare: Lawrence MacIntyre <lpz@ornl.gov>
Mottagare: Bugtraq (import) <29267>
Kommentar till text 10748424 av Earl Hood <earl@earlhood.com>
Sänt: 2003-09-26 19:57
Ärende: Re: base64
------------------------------------------------------------
From: Bennett Todd <bet@rahul.net>
To: Earl Hood <earl@earlhood.com>
Cc: bugtraq@securityfocus.com, MightyE <trash@mightye.org>,
Lawrence MacIntyre <lpz@ornl.gov>
Message-ID: <20030926170811.GC5537@rahul.net>
2003-09-25T19:46:36 Earl Hood:
> On September 25, 2003 at 11:30, Bennett Todd wrote:
> > There's a third method, which I think is rather better than either
> > of those. [canonicalize]
>
> You cannot do this for signed messages, therefore, you still
> need to either decode in all possible ways or drop the message
> (or the offending entity).
Or break the signature in the canonicalization.
Good catch. Lots of work will be needed to really completely solve
this, and different solutions will fit different security stances.
I think in terms of the security stances for corporations, with
particular focus on financial services firms. A very, very different
answer would be in order for e.g. an ISP.
For the kind of companies I work in, the very best solution would
(in my opinion!) be a canonicalizer that was smart enough to hold
off actually committing any rewrites until it finds something that's
ambiguous or dangerous, and that leaves notes describing what it did
and why.
Then when people get their mail whose sigs don't check, they get an
explanation of what needs fixing. Depending on the user they may
need to call a helpdesk to interpret the note and help them, or
their correspondent, to reconfig to fix the problem, but that's as
may be.
Also, in this sort of setting at least, you need very different
handling of inbound -vs- outbound messages. Inbound messages get
repaired --- or broken, in the case of digital sigs --- and then
sent on to their intended internal recipient. Outbound traffic gets
canonicalized if necessary, with commentary, gets malware replaced
with "evil badness used to be here, I yanked it", then gets bounced
back to the internal sender.
-Bennett
(10748375) /Bennett Todd <bet@rahul.net>/-----------
Bilaga (application/pgp-signature) i text 10748376
10748376 2003-09-26 13:08 -0400 /8 rader/ Bennett Todd <bet@rahul.net>
Importerad: 2003-09-26 19:50 av Brevbäraren
Extern mottagare: Earl Hood <earl@earlhood.com>
Extern kopiemottagare: bugtraq@securityfocus.com
Extern kopiemottagare: MightyE <trash@mightye.org>
Extern kopiemottagare: Lawrence MacIntyre <lpz@ornl.gov>
Mottagare: Bugtraq (import) <29268>
Bilaga (application/pgp-signature) till text 10748375
Ärende: Bilaga till: Re: base64
------------------------------------------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE/dHJ7HZWg9mCTffwRAr6RAKDNwxKYjQFa+blVWk15SegvTJ2p6ACfdeZO
VnPBJEh1Z/TFyfKUwiGswso=
=aCEM
-----END PGP SIGNATURE-----
(10748376) /Bennett Todd <bet@rahul.net>/-----------
10749047 2003-09-26 22:02 +0400 /32 rader/ 3APA3A <3APA3A@SECURITY.NNOV.RU>
Importerad: 2003-09-26 22:27 av Brevbäraren
Extern mottagare: Earl Hood <earl@earlhood.com>
Extern kopiemottagare: bugtraq@securityfocus.com
Extern kopiemottagare: Bennett Todd <bet@rahul.net>
Extern kopiemottagare: MightyE <trash@mightye.org>
Extern kopiemottagare: Lawrence MacIntyre <lpz@ornl.gov>
Externa svar till: 3APA3A@SECURITY.NNOV.RU
Mottagare: Bugtraq (import) <29288>
Kommentar till text 10748424 av Earl Hood <earl@earlhood.com>
Ärende: Re[2]: base64
------------------------------------------------------------
From: 3APA3A <3APA3A@SECURITY.NNOV.RU>
To: Earl Hood <earl@earlhood.com>
Cc: bugtraq@securityfocus.com, Bennett Todd <bet@rahul.net>,
MightyE <trash@mightye.org>, Lawrence MacIntyre <lpz@ornl.gov>
Message-ID: <622711973.20030926220256@SECURITY.NNOV.RU>
Dear Earl Hood,
Signature is applied to decoded message. If encoding is broken there
is no need to try to keep signature. Reassembling message is
really best solution. But there are another ways to bypass
message reassembling. Some of them can be found in
http://www.security.nnov.ru/advisories/content.asp
--Friday, September 26, 2003, 3:46:36 AM, you wrote to
bugtraq@securityfocus.com:
>> limits. Recode base64. Recode uuencoded chunks. Regularize
>> non-standard MIME.
EH> You cannot do this for signed messages, therefore, you still
EH> need to either decode in all possible ways or drop the message
EH> (or the offending entity).
EH> --ewh
--
~/ZARAZA
Èáî ôàêòû åñòü ôàêòû, è èçëîæåíû îíè ëèøü äëÿ òîãî, ÷òîáû èõ ïîíÿëè è â íèõ ïîâåðèëè. (Òâåí)
(10749047) /3APA3A <3APA3A@SECURITY.NNOV.RU>/(Ombruten)
10748494 2003-09-25 15:20 -0500 /29 rader/ Alun Jones <alun@texis.com>
Importerad: 2003-09-26 20:07 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29270>
Kommentar till text 10743801 av Bennett Todd <bet@rahul.net>
Ärende: RE: base64
------------------------------------------------------------
From: "Alun Jones" <alun@texis.com>
To: <bugtraq@securityfocus.com>
Message-ID: <20030926014436.29167.qmail@mail.securityfocus.com>
> -----Original Message-----
> From: Bennett Todd [mailto:bet@rahul.net]
> Sent: Thursday, September 25, 2003 10:30 AM
>
> Do all this canonicalization before the message hits your
> attachment type policy enforcement and malware scanner, so they only
> have to deal with the common forms that everybody handles the same.
With the obvious disadvantage that we're all reduced to using the
lowest-common-subset of functionality. Never mind inventing or
supporting new features, or adding international file naming support,
in your new email client, because the mail server will strip all of
that out, anyway. I don't think that's an appropriate answer.
Alun.
~~~~
--
Texas Imperial Software | Find us at http://www.wftpd.com or email
1602 Harvest Moon Place | alun@texis.com.
Cedar Park TX 78613-1419 | WFTPD, WFTPD Pro are Windows FTP servers.
Fax/Voice +1(512)258-9858 | Try our NEW client software, WFTPD Explorer.
(10748494) /Alun Jones <alun@texis.com>/--(Ombruten)
Kommentar i text 10748977 av Bennett Todd <bet@rahul.net>
10748977 2003-09-26 14:11 -0400 /31 rader/ Bennett Todd <bet@rahul.net>
Importerad: 2003-09-26 22:03 av Brevbäraren
Extern mottagare: Alun Jones <alun@texis.com>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29282>
Kommentar till text 10748494 av Alun Jones <alun@texis.com>
Ärende: Re: base64
------------------------------------------------------------
From: Bennett Todd <bet@rahul.net>
To: Alun Jones <alun@texis.com>
Cc: bugtraq@securityfocus.com
Message-ID: <20030926181118.GB6827@rahul.net>
2003-09-25T16:20:46 Alun Jones:
> > -----Original Message-----
> > From: Bennett Todd [mailto:bet@rahul.net]
> > Sent: Thursday, September 25, 2003 10:30 AM
> >
> > Do all this canonicalization before the message hits your
> > attachment type policy enforcement and malware scanner, so they only
> > have to deal with the common forms that everybody handles the same.
>
> With the obvious disadvantage that we're all reduced to using the
> lowest-common-subset of functionality.
Absolutely.
> Never mind inventing or supporting new features, or adding
> international file naming support, in your new email client,
> because the mail server will strip all of that out, anyway. I
> don't think that's an appropriate answer.
It's certainly not the appropriate answer for all settings.
Outlaw people from receiving email on Windows, and we can do away
with all this sludge.
-Bennett
(10748977) /Bennett Todd <bet@rahul.net>/-----------
Bilaga (application/pgp-signature) i text 10748978
10748978 2003-09-26 14:11 -0400 /8 rader/ Bennett Todd <bet@rahul.net>
Importerad: 2003-09-26 22:03 av Brevbäraren
Extern mottagare: Alun Jones <alun@texis.com>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <29283>
Bilaga (application/pgp-signature) till text 10748977
Ärende: Bilaga till: Re: base64
------------------------------------------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE/dIFGHZWg9mCTffwRAm3qAJ48BsWhIcVptlries/VuR7BHeI1fwCfcUYy
RjOmz7G6ioXSXzsvsItDxY0=
=s3HM
-----END PGP SIGNATURE-----
(10748978) /Bennett Todd <bet@rahul.net>/-----------