6796282 2001-07-24 23:36 +0300 /94 rader/ Stefan Laudat <stefan@mail.allianztiriac.ro>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-25 23:17 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18356>
Ärende: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Stefan Laudat <stefan@mail.allianztiriac.ro>
To: bugtraq@securityfocus.com
Message-ID: <20010724233639.A5717@allianztiriac.ro>
Hi Aleph,
Sent this already to you a couple of months ago but I presume it got
sucked into a quasar on its way to you.Maybe it gets lucky this time.
Looks like there are some problems in some of the most popular TCP/IP
stack implementations. I've found a kiddie-tool on the internet that
looks like it's rising some problems in a matter of CPU usage for
handling incoming UDP packets. Its initial aim was another one (read
the source) but accidentally it can be used for locking up machines.
You can try it from
http://rootshell.com/archive-j457nxiqi3gq59dv/199803/biffit.c
I'm not a TCP stack-writing guru but I presume the behaviour
described below is way beyond normal, as its results are quite
different depending on the OS used. Please don't bash me if I'm wrong.
Here are the reactions of major OS'es on the market on this simple
UDP flood. Be aware that the results are working on the quoted
kernels no matter if their specific ip-filtering mechanisms have UDP
filtering rules enabled or not and no matter if you really have
in.comsat running or not, you may change the destination port to
whatever and get the same result:
1. Linux 2.4.7 UP (pristine source, waiting for a new shiny Alan Cox
patch)
- system gets frozen after 3 seconds of flood on a gigabit
link. Same result at a 100Mbps. The top utility shows (at least as
long as it can) that system(kernel) gets 100% of the CPU in its march
to death. Same for Linux kernel 2.2.19.
2. Linux 2.4.7 SMP (same origin)
- the flood effect is distributed on the 2 CPUs (p3/1Ghz)
at a ratio of 30-40% per processor. Linux SMP is superb, it implements
load-sharing of everything, even DoS :)
3. Windows 2000 Server UP.
- the system graphs jump from 2% cpu usage (in a calm evening
with no ongoing backups and domain synchronizations) to approx. 35%
and holds it steady. The flood is performed via a Gigabit link. The
packet rate handling of win2k is wonderful, it even beats an OpenBSD
2.8. Kudos to MS guys, this one is a real hit. As I couldn't believe
my eyes I ran some applications on it (crunching queries on the local
MS SQL2k server etc) and I got timely-fashion responses.
4. Open BSD 2.8 UP, no patches applied
- top utility shows a 45% processor load and holds it steady.
The system is responsive (unlike Linux) and the apps are running okay.
5. Windows 2000 Professional UP.
- systems gets an extra 60% load average payload over its
initial one. It is still responsive to commands, doesn't choke a bit.
6. Windows ME UP
- this little bitch runs OpenGL apps under heavy UDP hammering
with no CPU cost ! Like nothing happened. Heilz to MS again.
7. Cisco IOS
- tested on Cisco 7513 (12.1(4)E, DCEF enabled), Cisco 2621
(12.2.1), Cisco Catalyst 35xxXL (12.0.5.2.XU and 12.0.5.1.XP). I'm
walking here on Linux-behaviour land. The CPU is burning at high load
averages from the start and I get no control over it. I can get the
result either directly attacking the router, either forwarding for a
host behind it. IOS plays the brave guy and gets it for anyone else
too.
8. SunOS 5.7 running on an UltraEnterprise 3500
- beautiful reaction. No CPU waving on this SMP system, looks
like nothing happened to it. I'll buy one for home when I'll be a
millionaire :)
I would like to hear some other results for other operating systems.
Greetz go as usual to Gore, my old one-eye pirate parrot and to his
fat wife that ruined his long glorious life.
P.S. [offtopic] can someone please write a detailed mail here to
explain how easily is to defend/log CodeRed with Snort and FlexResp
support? It would have been the only practical thing discussed about
it among tons of repetitive and identical advisories of 'security
experts' that solve nothing so far. I block hundreds of worms daily
this way without having a patched IIS. My recent post regarding this
was blocked already because the thread was (err...) "ended" by fate
or whatever so it wasn't relevant or even read.
--
Stefan Laudat
CCNA,CCAI
Senior Network Engineer
Allianz-Tiriac SA
"Let's call it an accidental feature."
-- Larry Wall
(6796282) /Stefan Laudat <stefan@mail.allianztiriac.ro>/(Ombruten)
Kommentar i text 6802330 av Michal Zalewski <lcamtuf@gis.net>
Kommentar i text 6802620 av trop <trop@softhome.net>
Kommentar i text 6802652 av Paul Sack <paulsack@mail.utexas.edu>
Kommentar i text 6806305 av Juergen P. Meier <jpm@jors.net>
Kommentar i text 6811353 av Pavlos Parissis <p_pavlos@otenet.gr>
6802330 2001-07-25 17:38 -0400 /59 rader/ Michal Zalewski <lcamtuf@gis.net>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 00:21 av Brevbäraren
Extern mottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18382>
Kommentar till text 6796282 av Stefan Laudat <stefan@mail.allianztiriac.ro>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Michal Zalewski <lcamtuf@gis.net>
To: Stefan Laudat <stefan@mail.allianztiriac.ro>
Cc: bugtraq@securityfocus.com
Message-ID: <Pine.LNX.4.21.0107251732400.747-100000@nimue.bos.bindview.com>
On Tue, 24 Jul 2001, Stefan Laudat wrote:
> /.../ looks like it's rising some problems in a matter of CPU usage
> for handling incoming UDP packets. Its initial aim was another one
> (read the source) but accidentally it can be used for locking up
> machines. You can try it from
> http://rootshell.com/archive-j457nxiqi3gq59dv/199803/biffit.c
>
> I'm not a TCP stack-writing guru but I presume the behaviour described
> below is way beyond normal, as its results are quite different
> depending on the OS used. Please don't bash me if I'm wrong.
Uh-huh. Tested it on Linux 2.2 and 2.4, can't confirm the problem. It
would be pretty strange, btw, since it simply generates normal UDP
packet, no black magic, really, and remote system, unless there's
comast service running, politely responds with 'ICMP destination port
unreachable', which is translated into 'Connection refused'.
Nothing magic about its behavior:
sendto(4, "test@0", 6, 0, {sin_family=AF_INET, sin_port=htons(512),
sin_addr=inet_addr("127.0.0.1")}}, 16) = -1 ECONNREFUSED (Connection
refused)
> 1. Linux 2.4.7 UP (pristine source, waiting for a new shiny Alan Cox patch)
> - system gets frozen after 3 seconds of flood on a gigabit link.
Maybe there's comsat service running? Or you made system too busy
handling I/O by flooding using 1 Gbit (I doubt it)...
> 3. Windows 2000 Server UP. - the system graphs jump from 2% cpu usage
> (in a calm evening with no ongoing backups and domain
> synchronizations) to approx. 35% and holds it steady.
Windows are usually impacted by high-ratio packet floods.
> The flood is performed via a Gigabit link. The packet rate handling of
> win2k is wonderful, it even beats an OpenBSD 2.8. Kudos to MS guys,
> this one is a real hit. As I couldn't believe my eyes I ran some
> applications on it (crunching queries on the local MS SQL2k server
> etc) and I got timely-fashion responses.
I believe you are actually testing link layer performance, PCI bus
speed and network cards, not operating systems ;)
--
_____________________________________________________
Michal Zalewski [lcamtuf@bos.bindview.com] [security]
[http://lcamtuf.coredump.cx] <=-=> bash$ :(){ :|:&};:
=-=> Did you know that clones never use mirrors? <=-=
(6802330) /Michal Zalewski <lcamtuf@gis.net>/(Ombruten)
Kommentar i text 6802621 av Kevin Day <toasty@temphost.dragondata.com>
Kommentar i text 6802627 av Stefan Laudat <stefan@mail.allianztiriac.ro>
Kommentar i text 6802628 av Cade Cairns <cairnsc@securityfocus.com>
Kommentar i text 6802658 av Stefan Laudat <stefan@mail.allianztiriac.ro>
Kommentar i text 6803161 av David LeBlanc <dleblanc@mindspring.com>
Kommentar i text 6806121 av Radu-Adrian Feurdean <raf@chez.com>
6802621 2001-07-26 16:48 -0500 /33 rader/ Kevin Day <toasty@temphost.dragondata.com>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 02:21 av Brevbäraren
Extern mottagare: Michal Zalewski <lcamtuf@gis.net>
Extern kopiemottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18392>
Kommentar till text 6802330 av Michal Zalewski <lcamtuf@gis.net>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Kevin Day <toasty@temphost.dragondata.com>
To: lcamtuf@gis.net (Michal Zalewski)
Cc: stefan@mail.allianztiriac.ro (Stefan Laudat),
bugtraq@securityfocus.com
Message-ID: <200107262148.f6QLmZL97633@temphost.dragondata.com>
>
> > The flood is performed via a Gigabit link. The packet rate handling of
> > win2k is wonderful, it even beats an OpenBSD 2.8. Kudos to MS guys,
> > this one is a real hit. As I couldn't believe my eyes I ran some
> > applications on it (crunching queries on the local MS SQL2k server
> > etc) and I got timely-fashion responses.
>
> I believe you are actually testing link layer performance, PCI bus speed
> and network cards, not operating systems ;)
>
Actually, you're probably entering a "livelock" situation. Packets
were coming in so fast that the interrupt handler is consuming all
your time. Alot of high speed network devices have special modes to
prevent this from happening. (Only interrupt when a certain number of
packets are in the FIFO, make sure the interrupt isn't asserted for
more than x% of the time, etc).
This is probably possible on 100MBit links on slow CPUs too.
Which network card are you using? I don't ever want to buy one. :)
--
Kevin Day
toasty@dragondata.com - kevin@stileproject.com
(6802621) /Kevin Day <toasty@temphost.dragondata.com>/(Ombruten)
6802627 2001-07-26 01:43 +0300 /53 rader/ Stefan Laudat <stefan@mail.allianztiriac.ro>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 02:23 av Brevbäraren
Extern mottagare: Michal Zalewski <lcamtuf@gis.net>
Extern kopiemottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18398>
Kommentar till text 6802330 av Michal Zalewski <lcamtuf@gis.net>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Stefan Laudat <stefan@mail.allianztiriac.ro>
To: Michal Zalewski <lcamtuf@gis.net>
Cc: Stefan Laudat <stefan@mail.allianztiriac.ro>,
bugtraq@securityfocus.com
Message-ID: <20010726014337.A31276@allianztiriac.ro>
> Uh-huh. Tested it on Linux 2.2 and 2.4, can't confirm the problem. It
> would be pretty strange, btw, since it simply generates normal UDP packet,
> no black magic, really, and remote system, unless there's comast service
> running, politely responds with 'ICMP destination port unreachable', which
> is translated into 'Connection refused'.
Hmm. How many seconds did you actually run that?
> Nothing magic about its behavior:
Did I mentioned it's magic? Guess not :-/
> Maybe there's comsat service running? Or you made system too busy handling
> I/O by flooding using 1 Gbit (I doubt it)...
As I said, NO.
> Windows are usually impacted by high-ratio packet floods.
Not this time.
> I believe you are actually testing link layer performance, PCI bus speed
> and network cards, not operating systems ;)
Believe it or not, I got a OpenBSD-2.9 current hanged up out there.
I'll test further systems.
What amazed me was different types of system reaction with different
drivers at different links.
>
> --
> _____________________________________________________
> Michal Zalewski [lcamtuf@bos.bindview.com] [security]
> [http://lcamtuf.coredump.cx] <=-=> bash$ :(){ :|:&};:
> =-=> Did you know that clones never use mirrors? <=-=
>
--
Stefan Laudat
CCNA,CCAI
Senior Network Engineer
Allianz-Tiriac SA
"Let's call it an accidental feature."
-- Larry Wall
(6802627) /Stefan Laudat <stefan@mail.allianztiriac.ro>/
Kommentar i text 6802624 av Michal Zalewski <lcamtuf@gis.net>
6802624 2001-07-25 18:49 -0400 /24 rader/ Michal Zalewski <lcamtuf@gis.net>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 02:22 av Brevbäraren
Extern mottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18395>
Kommentar till text 6802627 av Stefan Laudat <stefan@mail.allianztiriac.ro>
Sänt: 2001-07-27 02:23
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Michal Zalewski <lcamtuf@gis.net>
To: Stefan Laudat <stefan@mail.allianztiriac.ro>
Cc: bugtraq@securityfocus.com
Message-ID: <Pine.LNX.4.21.0107251847250.747-100000@nimue.bos.bindview.com>
On Thu, 26 Jul 2001, Stefan Laudat wrote:
> Believe it or not, I got a OpenBSD-2.9 current hanged up out there.
> I'll test further systems. What amazed me was different types of
> system reaction with different drivers at different links.
Maybe there's something wrong with your approach or test
environment. As long as there's no comsat or any other service
running on this port, and you do not actually trash your link layer,
there's absolutely no reason for this to succeed. It is normal,
system-generated UDP packet, and apparently, as expected, has no
impact on boxes I can reach from there...
--
_____________________________________________________
Michal Zalewski [lcamtuf@bos.bindview.com] [security]
[http://lcamtuf.coredump.cx] <=-=> bash$ :(){ :|:&};:
=-=> Did you know that clones never use mirrors? <=-=
(6802624) /Michal Zalewski <lcamtuf@gis.net>/(Ombruten)
6802628 2001-07-26 16:39 -0600 /28 rader/ Cade Cairns <cairnsc@securityfocus.com>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 02:23 av Brevbäraren
Extern mottagare: Michal Zalewski <lcamtuf@gis.net>
Extern kopiemottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18399>
Kommentar till text 6802330 av Michal Zalewski <lcamtuf@gis.net>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Cade Cairns <cairnsc@securityfocus.com>
To: Michal Zalewski <lcamtuf@gis.net>
Cc: Stefan Laudat <stefan@mail.allianztiriac.ro>,
<bugtraq@securityfocus.com>
Message-ID: <Pine.GSO.4.30.0107261637260.29142-100000@mail>
On Wed, 25 Jul 2001, Michal Zalewski wrote:
> Uh-huh. Tested it on Linux 2.2 and 2.4, can't confirm the problem. It
> would be pretty strange, btw, since it simply generates normal UDP packet,
> no black magic, really, and remote system, unless there's comast service
> running, politely responds with 'ICMP destination port unreachable', which
> is translated into 'Connection refused'.
After Stefan made his post to Bugtraq, I performed a few tests on
machines running Linux 2.2.14 and Linux 2.4.0. I wrote a simple test
program to send a large number of small messages to an arbitrary
serviceless port on the target machines.
I was able to reproduce the problem on a slower (400mhz) machine
running 2.4.0, it virtually stopped responding until the flood ended.
Cade Cairns
SecurityFocus
http://www.securityfocus.com/
(6802628) /Cade Cairns <cairnsc@securityfocus.com>/(Ombruten)
Kommentar i text 6803050 av Michal Zalewski <lcamtuf@gis.net>
6803050 2001-07-26 21:30 -0400 /30 rader/ Michal Zalewski <lcamtuf@gis.net>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 07:55 av Brevbäraren
Extern mottagare: Cade Cairns <cairnsc@securityfocus.com>
Extern kopiemottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18413>
Kommentar till text 6802628 av Cade Cairns <cairnsc@securityfocus.com>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Michal Zalewski <lcamtuf@gis.net>
To: Cade Cairns <cairnsc@securityfocus.com>
Cc: Stefan Laudat <stefan@mail.allianztiriac.ro>,
bugtraq@securityfocus.com Message-ID:
<Pine.LNX.4.21.0107262125470.747-100000@nimue.bos.bindview.com>
On Thu, 26 Jul 2001, Cade Cairns wrote:
> After Stefan made his post to Bugtraq, I performed a few tests on
> machines running Linux 2.2.14 and Linux 2.4.0. I wrote a simple test
> program to send a large number of small messages to an arbitrary
> serviceless port on the target machines. I was able to reproduce the
> problem on a slower (400mhz) machine running 2.4.0, it virtually
> stopped responding until the flood ended.
Try the same via loopback device - should not work. I believe this is
not Linux kernel UDP handling problem. It might be, as suggested, but
something between hardware and software, instead (like "IRQ
congestion"), and probably should work for everything - TCP, ICMP? Of
course I can be wrong - all I say is that I was not able to reproduce
this behavior in my test network, maybe because it is 10 Mbit, and
can't see any special reason why UDP attack should be more successful
than any other...
--
_____________________________________________________
Michal Zalewski [lcamtuf@bos.bindview.com] [security]
[http://lcamtuf.coredump.cx] <=-=> bash$ :(){ :|:&};:
=-=> Did you know that clones never use mirrors? <=-=
(6803050) /Michal Zalewski <lcamtuf@gis.net>/(Ombruten)
Kommentar i text 6806044 av Cade Cairns <cairnsc@securityfocus.com>
Kommentar i text 6806313 av <aland@striker.ottawa.on.ca>
6806044 2001-07-27 07:40 -0600 /26 rader/ Cade Cairns <cairnsc@securityfocus.com>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 18:35 av Brevbäraren
Extern mottagare: Michal Zalewski <lcamtuf@gis.net>
Extern kopiemottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18423>
Kommentar till text 6803050 av Michal Zalewski <lcamtuf@gis.net>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Cade Cairns <cairnsc@securityfocus.com>
To: Michal Zalewski <lcamtuf@gis.net>
Cc: Stefan Laudat <stefan@mail.allianztiriac.ro>,
<bugtraq@securityfocus.com>
Message-ID: <Pine.GSO.4.30.0107270734250.10291-100000@mail>
On Thu, 26 Jul 2001, Michal Zalewski wrote:
> Try the same via loopback device - should not work. I believe this is not
> Linux kernel UDP handling problem. It might be, as suggested, but
> something between hardware and software, instead (like "IRQ congestion"),
> and probably should work for everything - TCP, ICMP? Of course I can be
> wrong - all I say is that I was not able to reproduce this behavior in my
> test network, maybe because it is 10 Mbit, and can't see any special
> reason why UDP attack should be more successful than any other...
Confirmed. Of course, this could be a wide variety of other things,
maybe even the NIC driver. Oh well..
Cade Cairns
SecurityFocus
http://www.securityfocus.com/
(6806044) /Cade Cairns <cairnsc@securityfocus.com>/(Ombruten)
6806313 2001-07-27 10:53 -0400 /35 rader/ <aland@striker.ottawa.on.ca>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 19:53 av Brevbäraren
Extern mottagare: Michal Zalewski <lcamtuf@gis.net>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18434>
Kommentar till text 6803050 av Michal Zalewski <lcamtuf@gis.net>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: aland@striker.ottawa.on.ca
To: Michal Zalewski <lcamtuf@gis.net>
Cc: bugtraq@securityfocus.com
Message-ID: <E15Q8zW-0006gF-00@giles.striker.ottawa.on.ca>
Michal Zalewski <lcamtuf@gis.net> wrote:
> Try the same via loopback device - should not work. I believe this is not
> Linux kernel UDP handling problem. It might be, as suggested, but
> something between hardware and software, instead (like "IRQ congestion"),
> and probably should work for everything - TCP, ICMP?
At the last Linux Kernel Summit, Jamal Hadi Salim had a proposal for
speeding up packet handling in the 2.5 kernel. The issue is currently
that each packet coming into a network interface triggers an
interrupt. It's the interrupt servicing overhead that is slowing the
machine.
The proposal for 2.5 was to disable interrupts on an interface after
the first packet, and use other methods for noticing and grabbing the
later packets. There are other operating systems (QNX, etc) that do
this already.
Jamal's tests showed that removing this overhead drastically sped up
the network response, and removed much of the CPU overhead.
> Of course I can be wrong - all I say is that I was not able to
> reproduce this behavior in my test network, maybe because it is 10
> Mbit,
Implementations which appear to work under small loads may not scale
to higher loads.
Alan DeKok.
(6806313) / <aland@striker.ottawa.on.ca>/-----------
6802658 2001-07-26 01:48 +0300 /27 rader/ Stefan Laudat <stefan@mail.allianztiriac.ro>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 02:40 av Brevbäraren
Extern mottagare: Michal Zalewski <lcamtuf@gis.net>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18407>
Kommentar till text 6802330 av Michal Zalewski <lcamtuf@gis.net>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Stefan Laudat <stefan@mail.allianztiriac.ro>
To: Michal Zalewski <lcamtuf@gis.net>
Cc: bugtraq@securityfocus.com
Message-ID: <20010726014804.B31276@allianztiriac.ro>
> Uh-huh. Tested it on Linux 2.2 and 2.4, can't confirm the problem. It
> would be pretty strange, btw, since it simply generates normal UDP packet,
> no black magic, really, and remote system, unless there's comast service
> running, politely responds with 'ICMP destination port unreachable', which
> is translated into 'Connection refused'.
One extra thing I haven't underlined so well in my announce: cisco
routers (and as well as other ones maybe) start crawling even
forwarding the flood not being the target itself only. Looks like an
UDP handling problem for me :( I have managed to kill a 7513 Cisco
Router with DCEF enabled and loads of other speed hacks. Try it for
yourself :)
--
Stefan Laudat
CCNA,CCAI
Senior Network Engineer
Allianz-Tiriac SA
"Let's call it an accidental feature."
-- Larry Wall
(6802658) /Stefan Laudat <stefan@mail.allianztiriac.ro>/(Ombruten)
Kommentar i text 6806078 av Niels Bakker <niels=bugtraq@bakker.net>
6806121 2001-07-27 12:55 +0200 /48 rader/ Radu-Adrian Feurdean <raf@chez.com>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 18:56 av Brevbäraren
Extern mottagare: Michal Zalewski <lcamtuf@gis.net>
Extern kopiemottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18429>
Kommentar till text 6802330 av Michal Zalewski <lcamtuf@gis.net>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Radu-Adrian Feurdean <raf@chez.com>
To: Michal Zalewski <lcamtuf@gis.net>
Cc: Stefan Laudat <stefan@mail.allianztiriac.ro>,
bugtraq@securityfocus.com Message-ID:
<Pine.LNX.4.21.0107271246240.3329-100000@WormHole.Intra.ZEHC.Net>
On Wed, 25 Jul 2001, Michal Zalewski wrote:
> On Tue, 24 Jul 2001, Stefan Laudat wrote:
>
> > http://rootshell.com/archive-j457nxiqi3gq59dv/199803/biffit.c
>
> Uh-huh. Tested it on Linux 2.2 and 2.4, can't confirm the problem. It
> would be pretty strange, btw, since it simply generates normal UDP packet,
> no black magic, really, and remote system, unless there's comast service
> running, politely responds with 'ICMP destination port unreachable', which
> is translated into 'Connection refused'.
>
> > 1. Linux 2.4.7 UP (pristine source, waiting for a new shiny Alan Cox patch)
> > - system gets frozen after 3 seconds of flood on a gigabit link.
>
> Maybe there's comsat service running? Or you made system too busy handling
> I/O by flooding using 1 Gbit (I doubt it)...
Tested several times with 2.2 kernels (and in the past with 2.0). If
a logging firewall is used machine becomes unresponsive, but if the
flood does dot take much time, it recovers after the flood ends.
Without a logging firewall, the machine remains responsive, but
becomes much slower. This highly depends on teh packet rate, but on a
100Mbps link it is close to impossible to make it get frozen. Mainly
because packets get dropped.
>
> > 3. Windows 2000 Server UP. - the system graphs jump from 2% cpu usage
> > (in a calm evening with no ongoing backups and domain
> > synchronizations) to approx. 35% and holds it steady.
What about packet loss ?
Radu-Adrian Feurdean
mailto: raf@chez.com
----------------------------------------------------------------------------
The light at the end of the tunnel is the headlight of an approaching train.
(6806121) /Radu-Adrian Feurdean <raf@chez.com>/(Ombruten)
6802652 2001-07-25 16:06 -0500 /45 rader/ Paul Sack <paulsack@mail.utexas.edu>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 02:37 av Brevbäraren
Extern mottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18406>
Kommentar till text 6796282 av Stefan Laudat <stefan@mail.allianztiriac.ro>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Paul Sack <paulsack@mail.utexas.edu>
To: Stefan Laudat <stefan@mail.allianztiriac.ro>
Cc: <bugtraq@securityfocus.com>
Message-ID: <Pine.BSO.4.33.0107251557290.10347-100000@jefe.2y.net>
Yesterday at 11:36pm, Stefan Laudat expounded:
++ Looks like there are some problems in some of the most popular
TCP/IP
++ stack implementations. I've found a kiddie-tool on the internet
that
++ looks like it's rising some problems in a matter of CPU usage for
handling
++ incoming UDP packets. Its initial aim was another one (read the
source)
++ but accidentally it can be used for locking up machines.
Most UDP packets should be firewalled from the Internet.
This is only really useful if someone has access to the local network. Is
Linux/UP actually *locking* or just temporarily unresponsive? Also, it is
invalid to compare Windows ME running on $3000 hardware with Linux/*BSD
running on an old Pentium. Are you running all of this on the same
hardware? Obviously faster hardware is going to be affected less by a UDP
flood. How about the network cards?
I am suspicious that you are just comparing hardware, given that
different versions of W2K perform much differently in your
analysis. (You said the load was server: 35%, professional: 60%) I
somehow doubt that MS tuned the network stack so much on the
``server'' version & wouldn't do the same on the ``professional''
version.
I bet a Sun E10K with lots of NICs could flood the Sun UE3500 with
lots of NICs, but that probably doesn't mean that the Solaris 8
network stack is better than the Solaris 8 network stack; it's
because the E10K is faster.
-Paul Sack
ECE, UT Austin
--
Someone will try to honk your nose today.
(6802652) /Paul Sack <paulsack@mail.utexas.edu>/(Ombruten)
Kommentar i text 6802672 av Stefan Laudat <stefan@mail.allianztiriac.ro>
6802672 2001-07-26 01:59 +0300 /49 rader/ Stefan Laudat <stefan@mail.allianztiriac.ro>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 02:47 av Brevbäraren
Extern mottagare: Paul Sack <paulsack@mail.utexas.edu>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18410>
Kommentar till text 6802652 av Paul Sack <paulsack@mail.utexas.edu>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Stefan Laudat <stefan@mail.allianztiriac.ro>
To: Paul Sack <paulsack@mail.utexas.edu>
Cc: bugtraq@securityfocus.com
Message-ID: <20010726015959.C31276@allianztiriac.ro>
> Most UDP packets should be firewalled from the Internet.
Agree.
> This is only really useful if someone has access to the local network. Is
> Linux/UP actually *locking* or just temporarily unresponsive? Also, it is
> invalid to compare Windows ME running on $3000 hardware with Linux/*BSD
> running on an old Pentium. Are you running all of this on the same
> hardware? Obviously faster hardware is going to be affected less by a UDP
> flood. How about the network cards?
Identical network cards for Win2k, Linux SMP and OpemBSD processor
(Intel Pro 100). Linux was run on dual p3/1Ghz(SMP), Pentium2/400Mhz
and P3/800Mhz (UP). Windows 2000 was run on p3/1Ghz UP. I've made
tests with same results against Linux UP boxes running on Celeron/600
with 3com Vortex and realtek 8139 NICs. I've outlined that the result
is the same no matter if you hit via 1Gbit or 100Mbit.
> I am suspicious that you are just comparing hardware, given that different
> versions of W2K perform much differently in your analysis. (You said the
> load was server: 35%, professional: 60%) I somehow doubt that MS tuned the
> network stack so much on the ``server'' version & wouldn't do the same on
> the ``professional'' version.
Some of the Linux servers have just the same configuration with the
w2k servers. The reaction IS different. That's what amazes me. Also
WinME was run on a cheap p2/350 box with an old intel NIC. No
slowdown at all :(
> I bet a Sun E10K with lots of NICs could flood the Sun UE3500 with lots of
> NICs, but that probably doesn't mean that the Solaris 8 network stack is
> better than the Solaris 8 network stack; it's because the E10K is faster.
well then someone will clear all this stuff for me.
--
Stefan Laudat
CCNA,CCAI
Senior Network Engineer
Allianz-Tiriac SA
"Let's call it an accidental feature."
-- Larry Wall
(6802672) /Stefan Laudat <stefan@mail.allianztiriac.ro>/(Ombruten)
Kommentar i text 6805863 av Sean Hunter <sean@uncarved.com>
Kommentar i text 6806064 av Adrian Chadd <adrian@creative.net.au>
Kommentar i text 6806342 av Jarno Huuskonen <Jarno.Huuskonen@uku.fi>
6805863 2001-07-27 08:56 +0100 /106 rader/ Sean Hunter <sean@uncarved.com>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 17:46 av Brevbäraren
Extern mottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: Paul Sack <paulsack@mail.utexas.edu>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18420>
Kommentar till text 6802672 av Stefan Laudat <stefan@mail.allianztiriac.ro>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
I was entirely unable to duplicate the problem on my linux-2.4.7 box,
and this is why:
Firstly, I apply some rate throttling on incoming and outgoing
packets thussly...
#jump to connection throttling table
$IPTABLES -t filter -A INPUT -j in-throttle
echo -n " rate throttling: "
#syn flood limit
$IPTABLES -t filter -A in-throttle -j RETURN -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -m limit --limit 10/sec
$IPTABLES -t filter -A in-throttle -j rate-logdrop -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN
#rst flood limit
$IPTABLES -t filter -A in-throttle -j RETURN -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -m limit --limit 10/sec
$IPTABLES -t filter -A in-throttle -j rate-logdrop -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST
$IPTABLES -t filter -A in-throttle -j RETURN -p tcp
#icmp rate limit
$IPTABLES -t filter -A in-throttle -j RETURN -p icmp -m limit --limit 25/sec
$IPTABLES -t filter -A in-throttle -j RETURN -p udp -m limit --limit 25/sec
#log the flood
$IPTABLES -t filter -A in-throttle -j LOG -m limit --limit 10/min --log-prefix "FW-IN-THROTTLE: "
#enable this to debug inbound flood throttling
#$IPTABLES -t filter -A in-throttle -j RETURN
#normally this is the policy
$IPTABLES -t filter -A in-throttle -j DROP
...I do the same sort of thing outgoing. YMMV, I am not a qualified
doctor etc etc etc. You may have to tweak the limits to make them
suitable for your site, but they work for me.
The next reason is that I proc rate limit icmp traffic thussly:
net.ipv4.icmp_destunreach_rate = 50
net.ipv4.icmp_echoreply_rate = 50
net.ipv4.icmp_paramprob_rate = 50
net.ipv4.icmp_timeexceed_rate = 50
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
On most modern linux systems you can add these lines to
/etc/sysctl.conf and go "sysctl -p" to install them.
on others you have to tweak /proc by hand, doing:
echo 50 > /proc/sys/net/ipv4/icmp_destunreach_rate
etc etc.
Finally, you should really firewall all traffic (including, but not
limited to, biff) that you don't really really have to serve[1] and
that goes for udp and tcp. And don't ignore the loopback interface
or your local users may bite you.
Hope this helps
Sean
[1] Strangely, I need to open the biff port on the loopback interface
or I get packet logs. I haven't had the chance to track that down
yet.
On Thu, Jul 26, 2001 at 01:59:59AM +0300, Stefan Laudat wrote:
> > Most UDP packets should be firewalled from the Internet.
>
> Agree.
>
> > This is only really useful if someone has access to the local network. Is
> > Linux/UP actually *locking* or just temporarily unresponsive? Also, it is
> > invalid to compare Windows ME running on $3000 hardware with Linux/*BSD
> > running on an old Pentium. Are you running all of this on the same
> > hardware? Obviously faster hardware is going to be affected less by a UDP
> > flood. How about the network cards?
>
> Identical network cards for Win2k, Linux SMP and OpemBSD processor (Intel
> Pro 100). Linux was run on dual p3/1Ghz(SMP), Pentium2/400Mhz and P3/800Mhz
> (UP). Windows 2000 was run on p3/1Ghz UP. I've made tests with same results
> against Linux UP boxes running on Celeron/600 with 3com Vortex and realtek
> 8139 NICs. I've outlined that the result is the same no matter if you hit
> via 1Gbit or 100Mbit.
>
> > I am suspicious that you are just comparing hardware, given that different
> > versions of W2K perform much differently in your analysis. (You said the
> > load was server: 35%, professional: 60%) I somehow doubt that MS tuned the
> > network stack so much on the ``server'' version & wouldn't do the same on
> > the ``professional'' version.
>
> Some of the Linux servers have just the same configuration with the w2k
> servers. The reaction IS different. That's what amazes me. Also WinME was
> run on a cheap p2/350 box with an old intel NIC. No slowdown at all :(
>
> > I bet a Sun E10K with lots of NICs could flood the Sun UE3500 with lots of
> > NICs, but that probably doesn't mean that the Solaris 8 network stack is
> > better than the Solaris 8 network stack; it's because the E10K is faster.
>
> well then someone will clear all this stuff for me.
>
> --
> Stefan Laudat
> CCNA,CCAI
> Senior Network Engineer
> Allianz-Tiriac SA
>
> "Let's call it an accidental feature."
> -- Larry Wall
(6805863) /Sean Hunter <sean@uncarved.com>/(Ombruten)
Bilaga (application/pgp-signature) i text 6805865
Kommentar i text 6810355 av Sean Hunter <sean@uncarved.com>
6805865 2001-07-27 08:56 +0100 /10 rader/ Sean Hunter <sean@uncarved.com>
Importerad: 2001-07-27 17:46 av Brevbäraren
Extern mottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: Paul Sack <paulsack@mail.utexas.edu>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18421>
Bilaga (text/plain) till text 6805863
Ärende: Bilaga till: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE7YR6eAgJf0xrWsZwRAnaTAJwJL/aQL2tW12fibyNDqr3ezXgEYQCeJyWH
dIIO3vhqYCIHx3BxKepe4IQ=
=C3W1
-----END PGP SIGNATURE-----
(6805865) /Sean Hunter <sean@uncarved.com>/---------
6810355 2001-07-28 23:42 +0100 /28 rader/ Sean Hunter <sean@uncarved.com>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-29 03:52 av Brevbäraren
Extern mottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18454>
Kommentar till text 6805863 av Sean Hunter <sean@uncarved.com>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: "Sean Hunter" <sean@uncarved.com>
To: bugtraq@securityfocus.com
Message-ID: <20010728234246.A24408@uncarved.com>
Regular readers of this list may be amused to know that since this
message hit the list I have been subject to sustained attempts to
attack my host using the udp flood thingy (and other methods) from
many different source addresses. Before I got bored, I logged more
than 500 unique source addresses in less than an hour. I have also
been subjected to several port scans, some of whom forged the
addresses of some of the icann root nameservers as the source
addresses of their packets[1]. This attack has given me the perfect
chance to test out my firewall rules "in anger", and has shown that
the udp rate limiter detailed in my previous message works perfectly
(although I have made some tweaks since the original posting that
have improved its performance further).
I'd like to thank those who helped me test my firewall for their
interest, but the box is still perfectly usable and I'd appreciate it
if they could turn their attentions elsewhere.
Thanks
Sean
[1]I don't use the ICANN root, so I don't contact the rsc root
servers very often as you might imagine.
(6810355) /Sean Hunter <sean@uncarved.com>/(Ombruten)
6806064 2001-07-27 18:30 +0800 /67 rader/ Adrian Chadd <adrian@creative.net.au>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 18:41 av Brevbäraren
Extern mottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: Paul Sack <paulsack@mail.utexas.edu>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18424>
Kommentar till text 6802672 av Stefan Laudat <stefan@mail.allianztiriac.ro>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Adrian Chadd <adrian@creative.net.au>
To: Stefan Laudat <stefan@mail.allianztiriac.ro>
Cc: Paul Sack <paulsack@mail.utexas.edu>, bugtraq@securityfocus.com
Message-ID: <20010727183020.R79763@ewok.creative.net.au>
On Thu, Jul 26, 2001, Stefan Laudat wrote:
> > Most UDP packets should be firewalled from the Internet.
>
> Agree.
>
> > This is only really useful if someone has access to the local network. Is
> > Linux/UP actually *locking* or just temporarily unresponsive? Also, it is
> > invalid to compare Windows ME running on $3000 hardware with Linux/*BSD
> > running on an old Pentium. Are you running all of this on the same
> > hardware? Obviously faster hardware is going to be affected less by a UDP
> > flood. How about the network cards?
>
> Identical network cards for Win2k, Linux SMP and OpemBSD processor (Intel
> Pro 100). Linux was run on dual p3/1Ghz(SMP), Pentium2/400Mhz and P3/800Mhz
> (UP). Windows 2000 was run on p3/1Ghz UP. I've made tests with same results
> against Linux UP boxes running on Celeron/600 with 3com Vortex and realtek
> 8139 NICs. I've outlined that the result is the same no matter if you hit
> via 1Gbit or 100Mbit.
Guys, guys.
The realtek cards suck. If you don't believe me, try reading the
device driver code for them in FreeBSD. Bill Paul slightly rips into
their lame design. I use a couple at home in my doze machines because
they were lying about. Getting 100mbit is painful - I don't use the
top-line hardware in the doze machines.
> > I bet a Sun E10K with lots of NICs could flood the Sun UE3500 with lots of
> > NICs, but that probably doesn't mean that the Solaris 8 network stack is
> > better than the Solaris 8 network stack; it's because the E10K is faster.
>
> well then someone will clear all this stuff for me.
>
When you're seeing the PC lockup, run vmstat 1 on it.
See how many interrupts/context switches are happening a second.
I bet the INT levels are stupidly high.
Case in point: When trying out squid on a pair of IDE disks hooked up
to a linux-2.2 box, I noted that it was crawling. after running
vmstat for a while, it was obvious that the box was handling an
absolutely *stupid* amount of interrupts per second. Turning on DMA
fixed that.
Now, Gige. Id on't remember the details of the original post, but
if you've got a gige card in a Win server, I'm betting that the
basic TCP/UDP processing is occuring on the card, not on the box.
Depending how much work went into the driver (read: I bet its more
than the state of the gige drivers under free unices) they might
even be generating the connection refused replies *on the card*.
Adrian
--
Adrian Chadd Yeah, for me its (XML) like the movie Titanic.
<adrian@creative.net.au> Everybody loves it.
I want to be different, so I hate it.
--Duane Wessels
(6806064) /Adrian Chadd <adrian@creative.net.au>/(Ombruten)
6806305 2001-07-27 17:26 +0200 /52 rader/ Juergen P. Meier <jpm@jors.net>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-27 19:51 av Brevbäraren
Extern mottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Externa svar till: jpm@class.de
Mottagare: Bugtraq (import) <18433>
Kommentar till text 6796282 av Stefan Laudat <stefan@mail.allianztiriac.ro>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: "Juergen P. Meier" <jpm@jors.net>
To: Stefan Laudat <stefan@mail.allianztiriac.ro>
Cc: bugtraq@securityfocus.com
Message-ID: <20010727172629.A13637@fm.rz.fh-muenchen.de>
> http://rootshell.com/archive-j457nxiqi3gq59dv/199803/biffit.c
>
> 1. Linux 2.4.7 UP (pristine source, waiting for a new shiny Alan Cox patch)
> - system gets frozen after 3 seconds of flood on a gigabit link.
> Same result at a 100Mbps. The top utility shows (at least as long as it can)
> that system(kernel) gets 100% of the CPU in its march to death. Same for
> Linux kernel 2.2.19.
2.4.6 (modular, unpatched, selfcompiled) on an old P133:
biffit against loopback: 99% cpu(system), no slowdown, system
responds normaly. (no slowdown) biffit against eth0: same
effect. (doh, cause linux sends it over loopback)
Biffit from a PIII/600 FDX 100mbit connected: same as above.
in the later case: my ssh connection to that system (going through
the same nic that was target) became a bit sluggish. Console access
showed no impact.
It obviously just consumes idle time. Interupt load was not very high.
(ping -f is much worse for interrupts)
Hardware:
Board: ASUS p55tp4n (Intel FX chipset) CPU: P133 (with F00F bug) RAM:
64mb eth0: RealTek RTL8139 Fast Ethernet at 0xc480b000,
00:00:cb:11:22:33, IRQ 11 eth0: Identified 8139 chip type
'RTL-8139C' (nothing else on IRQ11) running squid, a webserver, 3 ssh
connections, and having several iptables rules (those udp packets
matched ACCEPT-rule #2 (loopback case) and rule #4 on input chain)
no SMP.
> I would like to hear some other results for other operating systems.
Windows 98 (running on the P3/600):
25% load. no side effects
Juergen
--
Juergen P. Meier
(6806305) /Juergen P. Meier <jpm@jors.net>/(Ombruten)
Kommentar i text 6807893 av Keith Warno <keith.warno@valaran.com>
6807893 2001-07-27 18:18 -0400 /47 rader/ Keith Warno <keith.warno@valaran.com>
Sänt av: joel@lysator.liu.se
Importerad: 2001-07-28 05:21 av Brevbäraren
Extern mottagare: jpm@class.de
Extern kopiemottagare: Stefan Laudat <stefan@mail.allianztiriac.ro>
Extern kopiemottagare: bugtraq@securityfocus.com
Mottagare: Bugtraq (import) <18450>
Kommentar till text 6806305 av Juergen P. Meier <jpm@jors.net>
Ärende: Re: UDP packet handling weird behaviour of various operating systems
------------------------------------------------------------
From: Keith Warno <keith.warno@valaran.com>
To: jpm@class.de
Cc: Stefan Laudat <stefan@mail.allianztiriac.ro>,
bugtraq@securityfocus.com
Message-ID: <3B61E899.64E5071@valaran.com>
"Juergen P. Meier" wrote:
>
> > http://rootshell.com/archive-j457nxiqi3gq59dv/199803/biffit.c
> >
> > 1. Linux 2.4.7 UP (pristine source, waiting for a new shiny Alan Cox patch)
> > - system gets frozen after 3 seconds of flood on a gigabit link.
> > Same result at a 100Mbps. The top utility shows (at least as long as it can)
> > that system(kernel) gets 100% of the CPU in its march to death. Same for
> > Linux kernel 2.2.19.
>
> 2.4.6 (modular, unpatched, selfcompiled) on an old P133:
>
> biffit against loopback: 99% cpu(system), no slowdown, system
> responds normaly. (no slowdown)
> biffit against eth0: same effect. (doh, cause linux sends it over loopback)
Confirmed, with kernel 2.4.7 (unpatched, selfcompiled, modular).
~99% CPU usage, but no slowdown. Although the hardware is quite
different -- proc: Thunderbird 1000 ram: 384 MB mobo: MSI MS6340
micro atx, VIA KT 133 NIC: LinkSys LNE100TX v4.1 (using
kernel-distributed tulip driver, not that from Scyld) (eth0: ADMtek
Comet rev 17) -- altho this is irrelevent isn't it.
I've been using 2.4.whatever for only the past couple of weeks.
Maybe I'm missing something. I don't run any UDP services whatsoever
and would never run comsat under any circumstance. So, why does the
sendto() in biffit.c not fail when sending to localhost? When I boot
back to 2.2.19 and try the same thing, I get what I expect:
Connection refused. Hrmm. Unless I missed something in the kernel
docs regarding loopback behavior, the displayed 2.4.7 behavior seems
like a Bad Thing.
kw
--
| Keith Warno cell: +1 609-209-5800
| http://www.valaran.com/ work: +1 609-716-7200 x243
| Valaran Corporation Penguin Guy SMS : kw-mobile@valaran.com
+--------------------------------------------------------------//
(6807893) /Keith Warno <keith.warno@valaran.com>/(Ombruten)