5054580 2000-05-02  00:34  /1864 rader/ Postmaster
Mottagare: Bugtraq (import) <10691>
Ärende: Re: Source code to mstream, a DDoS tool
------------------------------------------------------------
Approved-By: aleph1@SECURITYFOCUS.COM
Delivered-To: bugtraq@lists.securityfocus.com
Delivered-To: bugtraq@securityfocus.com
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Message-ID:  <Pine.GUL.4.21.0005011351460.11047-100000@red7.cac.washington.edu>
Date:         Mon, 1 May 2000 14:08:15 -0700
Reply-To: Dave Dittrich <dittrich@CAC.WASHINGTON.EDU>
Sender: Bugtraq List <BUGTRAQ@SECURITYFOCUS.COM>
From: Dave Dittrich <dittrich@CAC.WASHINGTON.EDU>
X-To:         auscert@auscert.org.au, cert@cert.org, ciac@llnl.gov 
             submissions@packetstorm.securify.com, bugtraq@securityfocus.com 
             2600-list@wiretapped.net, contact@hackernews.com 
             webmaster@2600.com, vuln-dev@securityfocus.com 
             Incidents Mailing List <INCIDENTS@securityfocus.com>,
oth@2600.com To: BUGTRAQ@SECURITYFOCUS.COM In-Reply-To:
<200004291748.TAA13203@lobeda.jena.thur.de>

==========================================================================

       The "mstream" distributed denial of service attack tool

==========================================================================

May 1, 2000
Copyright (C) 2000. All rights reserved.

David Dittrich
University of Washington
<dittrich@cac.washington.edu>

George Weaver
Pennsylvania State University
<weaver@gabriel.nso.psu.edu>

Sven Dietrich
NASA Goddard Space Flight Center
<spock@netsec.gsfc.nasa.gov>

Neil Long
Oxford University
<neil.long@computing-services.oxford.ac.uk>


Introduction
------------

The following is an analysis of "mstream", a distributed denial of
service (DDoS) attack tool, based on the source code of "stream2.c", a
classic point-to-point DoS attack tool [12].

There are currently seven major code bases of recognized DDoS tools
witnessed in use "in the wild" for executing DDoS attacks:

	trinoo [03]
	Tribe Flood Network (TFN) [04]
	Tribe Flood Network 2000 (tfn2k) [06]
	stacheldraht/stacheldrahtV4 [05]
	stacheldraht v2.666
	shaft [07]
	mstream

mstream is more primitive than any of the other DDoS tools.
Examination of reverse engineered and recovered C source code reveals
the program to be in early development stages, with numerous bugs and
an incomplete feature set compared with any of the other listed
tools.  The effectiveness of the stream/stream2 attack itself,
however, means that it will still be disruptive to the victim (and
agent) networks even with an attack network consisting of only a hand
full of agents.

(The source code for mstream was anonymously posted to the
vuln-dev@securityfocus.com and BUGTRAQ email lists on April 29,
2000. [20] As a result, this analysis is being released in a bit less
polished and complete form that originally intended, but will
hopefully allow incident response teams and vendors to save some time
in developing their responses.  Any errors we didn't catch are likely
a result of this rush.)

The reader is advised that modification of the source code can and
would change any of the details of this analysis, such as prompts,
passwords, commands, TCP/UDP port numbers, or supported attack
methods, signatures, and features.  In fact, the values of
communication ports have already been seen to differ between
published and "in the wild" code.

[Note also that throughout this analysis, actual nicks,
site names, and IP addresses have been sanitized.]

The handler/agent terminology used in this analysis was developed at
the CERT Distributed System Intruder Tools workshop held in November
1999, and will be used in this analysis.  It is highly recommended
that the CERT workshop report [01] be read first, as well as
background and supporting information on other DDoS tools [08]:

	http://www.cert.org/reports/dsit_workshop.pdf
	http://staff.washington.edu/dittrich/misc/ddos/

An mstream agent was discovered in late April 2000 on a compromised
Linux system at a major university.  This system was identified to be
flooding packets using forged source addresses, targeted at over a
dozen IP addresses.

Since RFC 2267 style egress filtering [13] was employed on this
network, and all 32 bits of the source addresses were forged, only an
extremely small number of attack packets (matching the internal
network blocks) could have managed to leave the agent's network.  The
traffic did, however, cause the router (which served 18 subnets) to
become non-responsive.  This means that sites that do egress
filtering may still suffer from these attacks themselves, even if the
intended "victim" receives fewer packets than the attacker(s)
intended.  The lesson here is that there is no "quick fix" to DDoS in
the form of simple technical filtering solutions. [The router
manufacturer was briefed and is currently working on identifying the
cause and fixes.]

Another side-effect of taking egress filtering down one level to that
of internal subnets is that rejected packets will not make it past
the router doing the filtering, so the effects of bandwidth
consumption or router disruption will not be felt above the level of
the router doing the filtering (or being saturated).  This means a
border router based IDS (e.g., net flows), or one outside your
borders on a DMZ or upstream ISP's network, will not identify the
attempted attacks.  Unless you are monitoring the routers themselves,
only user complaints would tip you off to an attack originating from
your network.  It also means that packet level analysis is made more
difficult, as it must be done *in front* of the router doing the
filtering in order to capture all packets.  This puts an added burden
on network engineers or incident responders to do packet level
dumping, on site in a router closet, in order to know with a high
degree of reliability what is going on.

A major problem in forensic analysis of these attacks -- whether
successful or blocked by egress filters -- is having policies and
procedures in place to facilitate packet level logging.  At least one
site that was a victim of early February 2000 DoS attacks on eCommerce
sites was not prepared for this kind of analysis, and to this day is
not aware of what attack tool was used against them.  They only knew
their routers were crashing and relied on their upstream provider to
determine how to block packets to restore throughput.  This results in
a major roadblock to investigation by law enforcement, who have been
criticized heavily by some for their slow response (it is a
non-trivial problem to perform an investigation of a DoS or DDoS
attack with nothing but router or browser response behavioral
observations to go on.)  A description of the details of the attack on
Yahoo! was published on Packet Storm Security's web site [11], but
not many other details were available.  (This was also pointed out by
Anonymous [20] in his/her post.)

Unlike the high volume mass intrusions witnessed with trinoo, TFN,
and stacheldraht in 1999, this particular mstream network appears to
be in the very early stages of code development and to have been set
up by hand (on both handler and agent systems) with a slightly
modified Linux rootkit version 4 [10] to conceal the presence of the
intruder's activity.  For a description of intrusion and concealment
on an agent system, see Appendix D.  For a description of intrusion
and concealment on a handler system, see Appendix E.


The network: client(s)-->handler(s)-->agent(s)-->victim(s)
------------------------------------------------------------

The mstream network, like trinoo and shaft, is made up of one or more
handlers ("master.c") and a large set of agents ("server.c").
Attacker to handler communication is at present unencrypted over TCP,
with handler <-> agent communication unencrypted over UDP.

An mstream network would look like this:

                  +----------+           +----------+
                  | attacker |           | attacker |
                  +----------+           +----------+
                       |                      |
        . . . --+------+---------------+------+----------------+-- . . .
                |                      |                       |
                |                      |                       |
          +-----------+          +-----------+           +-----------+
          |  handler  |          |  handler  |           |  handler  |
          +-----------+          +-----------+           +-----------+
                |                      |                       |
                |                      |                       |
. . . ---+------+-----+------------+---+--------+------------+-+-- . . .
         |            |            |            |            |
         |            |            |            |            |
     +-------+    +-------+    +-------+    +-------+    +-------+
     | agent |    | agent |    | agent |    | agent |    | agent |
     +-------+    +-------+    +-------+    +-------+    +-------+



Communication
-------------

    Attacker to handler(s):	6723/tcp  (in published source)
				15104/tcp ("in the wild")
				12754/tcp (in recovered source)
    Agent to Handler(s):	9325/udp  (in published source)
				6838/udp  ("in the wild")
    Handler to agent(s):	7983/udp  (in published source)
				10498/udp ("in the wild")

Remote control of the mstream handler is accomplished via a TCP
connection to port 6723/tcp (or 15104/tcp, or 12754/tcp, or...).

The handler expects commands to be contained entirely in the data
payload of a single TCP packet, not broken up character by character
in a stream.  This means that "telnet" cannot be used to control a
handler, but instead some other client program must be used to buffer
the command line before sending (e.g., a special command shell or port
redirector, netcat [19], etc. -- no special client was included in the
source posted on Security Focus [20]).

The traffic over this connection is not encrypted (although it has
been shown in stacheldraht that adding a Blowfish block cipher is not
difficult.)  Command lines are space separated argument lists.

Like trinoo, communication between the handler(s) and agent(s) is
accomplished using UDP datagrams.  Agent commands are slash ("/")
separated argument lists, with some multi-item arguments being colon
(":") separated lists.

[It is believed that the difference between attacker <-> handler
communication port numbers between "lsof" output on a system running
an active handler and that in the recovered source code is due to
version differences.  That is, the recovered source may have been an
older version than the source used to compile the handler actually
running at the time.  Both ports are identified here, although they
are trivial to change to something else (and were found to be
different in the published source [20]).]

The examined code limits the maximum number of connected attackers to
3.  This may be a protective measure, or possibly a redundant access
measure in case one or more systems used as intermediaries by the
attacker are discovered or otherwise taken out of service.

After connecting, the user must supply the proper password (default
is "N7%diApf!" in the recovered code, and "sex" in the published code
[20]).  If the proper password is not given, all currently connected
users are notified of the attempt and the connection is dropped.  If
the proper password is given, all currently connected users are
informed of the new session and the user is presented with a "> "
prompt.


Handler Commands
----------------

The handler commands consist of up to 3 space-delimited fields.

If a connected attacker does not enter a command within 420 seconds,
the connection is terminated.  (It is assumed "420" was not chosen
simply because 7 minutes is an ideal timeout value. ;)

The handler command set is:

help
	Prints the following:

        Available commands:
        stream		stream attack !
        servers		Prints all known servers.
        ping		ping all servers.
        who		tells you the ips of the people logged in
        mstream		lets you stream more than one ip at a time

servers
	List all currently known agents.

who
	Shows the currently connected users.

ping
	Identify remaining active agents.  Sends the command "ping" to
	all known agents and reports to the connected users as each
	"pong" reply is received.

stream <hostname> <seconds>
	Begin an attack against a single host, for the specified
	duration.  The handler resolves the hostname to an IP address
	and sends the command "mstream/arg1:arg1/arg2" to all agents,
	where "arg1" is the resolved hosts' IP address twice with a
	colon between (this simplifies argument parsing in the agent)
	and "arg2" is the duration in seconds.

mstream <ip1:ip2:ip3:...> <seconds>
	Begin an attack against multiple IP addresses, for the specified
	duration.  The handler sends the command "mstream/arg1/arg2" to
	all agents, where "arg1" is the list of colon separated IP
	addresses, and "arg2" is the duration in seconds.  Also for
	simplicity, in this command, there is no host name resolution
	(i.e., you MUST specify all targets by a properly formed colon
	separated list of IP addresses)

quit
	Terminates the attacker's connection to the handler.


Agent Commands
--------------

The handler communicates to agents using string based commands
in the data portion of UDP packets.  These commands are not encrypted
(although this, too, can easily be changed.)

There are only three agent commands currently.  Commands are either
a simple string, or a slash ("/") separated command and argument list.

ping
	Replies to IP address that sent this packet with "pong".

stream/IP/seconds
	Starts streaming at IP address for specified duration in
	seconds.

mstream/IP1[:IP2[:IPN]]/seconds
	Starts streaming at all of the colon separated list of IP
	addresses for specified duration in seconds.

Even though the agent "in the wild" had two options for accepting
DDoS commands, namely "stream" and "mstream" as in the published
source, only the mstream command is used in the handler to agent
protocol. A simple "stream 192.168.0.100 10" command to the handler
sends the powerful command "mstream/192.168.0.100:192.168.0.100/10"
to the agent, when in fact a simple "stream/192.168.0.100/10" should
be generated. It is not clear why this was done, but it does look
like simple algorithms are used for command parsing, so this might
just indicate a "quick and dirty" development process.


Password protection
-------------------

The handler is password protected, to prevent trivial takeover of
the network handler.  The password is not encrypted, just a string
that is compared against the data paylod of the initial packet as-is.

It should be explicitly noted here again that this program has a
feature not found in other DDoS tools, which informs all connected
users of access, sucessful or not, to the handler(s) by competing
parties (black hat or white hat).  Thus it does not matter that you
can identify the password string in the binary, since you can't use
it without detection (and can't simply hijack the TCP session,
either, because of the command buffering described in the
Communication section.)

There is no password protection of handler <-> agent communication,
but that isn't surprising. As was seen with trinoo, a password in
clear text is not much of a defense and is trivially attacked by
sniffing network traffic.


Fingerprints
------------

As mentioned above, command strings between the handler(s) and
agent(s) is visible in packet flows.

Visible strings in the agent (in two truncated columns to save space)
are:

------------------------------------------------------------------------------
ELF				    mstream
/lib/ld-linux.so.2		    ping
GNU				    pong
__gmon_start__			    fork
libc.so.6			    init.c
random				     . . .
getpid				    server.c
perror				    strchr@@GLIBC_2.0
getuid				    packet
malloc				    getpid@@GLIBC_2.0
recvfrom			    _DYNAMIC
socket				    _etext
bind				    __register_frame_info@@GLIBC_2.0
inet_addr			    recvfrom@@GLIBC_2.0
__deregister_frame_info		    _fp_hw
setsockopt			    perror@@GLIBC_2.0
rand				    fork@@GLIBC_2.0
strncmp				    sock
strncpy				    cksum
sendto				    random@@GLIBC_2.0
strtok				    _init
fork				    malloc@@GLIBC_2.0
memset				    getppid@@GLIBC_2.0
srand				    sendto@@GLIBC_2.0
getppid				    __deregister_frame_info@@GLIBC_2.0
time				    setsockopt@@GLIBC_2.0
htons				    time@@GLIBC_2.0
exit				    _start
atoi				    forkbg
_IO_stdin_used			    strlen@@GLIBC_2.0
__libc_start_main		    stream
strlen				    strncmp@@GLIBC_2.0
strchr				    inet_addr@@GLIBC_2.0
__register_frame_info		    __bss_start
free				    main
GLIBC_2.0			    __libc_start_main@@GLIBC_2.0
PTRh				    data_start
QVh0				    bind@@GLIBC_2.0
Ph%				    getuid@@GLIBC_2.0
PhG				    _fini
WVS				    s_in
[^_				    srand@@GLIBC_2.0
WVS				    nlstr
j(j				    exit@@GLIBC_2.0
j h				    atoi@@GLIBC_2.0
j(h				    _edata
j h				    in_cksum
j(h				    _GLOBAL_OFFSET_TABLE_
[^_				    free@@GLIBC_2.0
131.247.208.191			    _end
129.79.20.202			    htons@@GLIBC_2.0
socket				    send2master
bind				    memset@@GLIBC_2.0
setsockopt			    strncpy@@GLIBC_2.0
newserver			    _IO_stdin_used
stream				    strtok@@GLIBC_2.0
__data_start			    __gmon_start__
socket@@GLIBC_2.0                   rand@@GLIBC_2.0
------------------------------------------------------------------------------

Visible strings in the handler are:

------------------------------------------------------------------------------
% strings -n 3 master  		    Available commands:
socket				    stream
bind				    stream attack !
listen				    servers
setsockopt			    Prints all known servers.
fcntl				    ping
You're too idle !		    ping all servers.
Connection from %s		    who
newserver			    tells you the ips of the people log
New server on %s.		    mstream
pong				    lets you stream more than one ip at
Got pong number %d from %s	    who
%s has disconnected (not auth'd): % Currently Online:
Invalid password from %s.	    Socket number %d
Password accepted for connection fr [%s]
Lost connection to %s: %s	    ping
stream				    Pinging all servers.
Usage: stream <hostname> <seconds>  mstream
Unable to resolve %s.		    Usage: mstream <ip1:ip2:ip3:...> <s
stream/%s/%s			    MStreaming %s for %s seconds.
Streaming %s for %s seconds.	    mstream/%s/%s
quit				    fork
%s has disconnected.		    Forked into background, pid %d
servers				    Caught SIGHUP, ignoring.
Server file doesn't exist, creating Caught SIGINT, ignoring.
The following ips are known servers Segmentation Violation, Exiting cle
help				    Caught unknown signal, This should
commands
------------------------------------------------------------------------------

The agent (named "rpc.wall" on this system -- this same name was used
for the handler as well) is seen with "lsof" as follows:

------------------------------------------------------------------------------
COMMAND    PID     USER   FD   TYPE     DEVICE    SIZE      NODE NAME
rpc.wall   588     root  cwd    DIR        3,2    1024         2 /
rpc.wall   588     root  rtd    DIR        3,2    1024         2 /
rpc.wall   588     root  txt    REG        3,3   17016     15765 /usr/bin/rpc.wall
rpc.wall   588     root  mem    REG        3,2  342206     30771 /lib/ld-2.1.1.so
rpc.wall   588     root  mem    REG        3,2 4016683     30789 /lib/libc-2.1.1.so
rpc.wall   588     root    0u   CHR        5,1              4952 /dev/console
rpc.wall   588     root    1w  FIFO        0,0               646 pipe
rpc.wall   588     root    2w  FIFO        0,0               647 pipe
rpc.wall   588     root    3u  IPv4        656               UDP *:10498
rpc.wall   588     root    4u  IPv4        657               UDP *:1044
rpc.wall   588     root    5u  IPv4        658               UDP *:1045
rpc.wall   588     root    6u   raw                        30219 00000000:00FF->00000000:0000 st=07
rpc.wall   588     root    7r  FIFO        0,0               648 pipe
rpc.wall   588     root    8u   raw                        30241 00000000:00FF->00000000:0000 st=07
rpc.wall   588     root    9u   CHR        5,1              4952 /dev/console
rpc.wall   588     root   10u  IPv4      30244               UDP *:1051
rpc.wall   588     root   11u   raw                        30245 00000000:00FF->00000000:0000 st=07
rpc.wall   588     root   21w  FIFO        0,0               648 pipe
------------------------------------------------------------------------------

Bugs in the source code for both handler and agent result in an
increasing number of raw sockets and UDP sockets in the agent (three
each are were witnessed on this agent), and an increasing number of
open file handles and UDP sockets in the handler (hundreds were shown
by Andrew Korty).  [This is no doubt an indication that mstream is in
the early development stages, so this signature should not be counted
on to identify a handler or agent running on a system.]

When an agent first starts up, it sends a "newserver" command to
the list of default handlers compiled into it, as seen here with
tcpdump:

----------------------------------------------------------------------------
00:04:38.530000 192.168.0.20.1081 > 192.168.0.100.6838: udp 9
0x0000   4500 0025 ef75 0000 4011 098a c0a8 0014 E..%.u..@.......
0x0010   c0a8 0064 0439 1ab6 0011 2b63 6e65 7773 ...d.9....+cnews
0x0020   6572 7665 7200 0000 0000 0000 0000      erver.........
----------------------------------------------------------------------------

If a rootkit is in place (as it was on both handler and agent
systems), you cannot trust the standard operating system
commands to show you the running handler or agent, or their
network connections.  The main lesson to be learned from rootkits is
that a large percentage of Unix system administrators will NOT be
skilled enough to get around rootkits.  This means a couple things.

First, and fundamentally, intruders will tend to have an even greater
advantage over unskilled system administrators.  It is becoming ever
more important that systems administrators -- Unix, NT, whatever --
have training as a primary task, not a luxury or burden to be
avoided.  The moment of a security incident is NOT the proper time to
catch up on months of missed learning, and causes undue pressure to
take shortcuts to get the system back up quickly (usually making
things MUCH worse).

Second, incident response and forensic investigation may be made more
difficult, if not impossible, as the simple "solution" that the
unskilled Unix administrator will take is to give up an just
re-install the operating system.  This ill-advised choice of action
destroys any evidence that may exist on the system and sets the
system up for a subsequent intrusion because the same security
precautions they did not take before the incident will usually not be
taken this time either.  All too often, this action is taken without
first seeking advice and before incident response teams are able to
notify the administrator and assist them in taking the correct steps.
All system administrators are urged to take the time now to prepare
to deal with rootkits [10].

As mentioned above, if an agent receives a UDP packet on port
10498/udp, containing the string "ping" as its data payload (and if it
is not actively streaming at the time), it will reply to the sending
system with a UDP packet on port 6838/udp with the string "pong" as
its data payload. (The trailing zeroes are just a tcpdump artifact.
The payload is clearly 4 bytes.)

----------------------------------------------------------------------------

00:05:16.457239 192.168.0.100.65364 > 192.168.0.20.10498: udp 5
0x0000   4500 0021 f412 0000 4011 04f1 c0a8 0064 E..!....@......d
0x0010   c0a8 0014 ff54 2902 000d 6ce3 7069 6e67 .....T)...l.ping
0x0020   0a                                             .

00:05:16.458214 192.168.0.20.1083 > 192.168.0.100.6838: udp 4
0x0000   4500 0020 ef8c 0000 4011 0978 c0a8 0014 E.......@..x....
0x0010   c0a8 0064 043b 1ab6 000c 8045 706f 6e67 ...d.;.....Epong
0x0020   0000 0000 0000 0000 0000 0000 0000       ..............
----------------------------------------------------------------------------

This sequence allows signature matching with programs like "ngrep"
[14], "snort" [18] (see Appendix B for snort rules), or scanning for
idle agents using "rid" [15] (see Appendix C for a RID template).

The ngrep command string to detect these packets would be:

	# ngrep "p[oi]ng" udp port 6838 or udp port 10498

(You will need to tailor this command, or add the other ports listed
above from the published code, to decrease the chance of false
negatives.)

Attack packets have a fixed size of 40 bytes per packet, which may be
on purpose to evade large packet triggers that exist on some IDSs.

The stream2.c attack floods the victim with TCP ACK packets, using
forged source addresses generated by random() (i.e., any or all
of the four octets will occasionally be zero) and incrementing source
port and sequence numbers, as seen in this code snippet:

------------------------------------------------------------------------------
 . . .
    for(i=0;;++i) {
    cksum.pseudo.saddr = packet.ip.ip_src.s_addr = random();
       ++packet.ip.ip_id;
       ++packet.tcp.th_sport;
       ++packet.tcp.th_seq;

       if (!dstport)
          s_in.sin_port = packet.tcp.th_dport = rand();
 . . .
------------------------------------------------------------------------------

During an attack, the following packet signature would be observed
(as seen by tcpdump using a recovered agent binary):

----------------------------------------------------------------------------
01:39:24.701083 192.168.0.2.65527 > 192.168.0.20.10498: [bad udp cksum 3100!]
udp 24 (ttl 64, id 886)
0x0000   4500 0034 0376 0000 4011 f5dc c0a8 0002 E..4.v..@.......
0x0010   c0a8 0014 fff7 2902 0020 556c 7374 7265 ......)...Ulstre
0x0020   616d 2f31 3932 2e31 3638 2e30 2e31 3030 am/192.168.0.100
0x0030   2f31 300a                                      /10.

01:40:10.132724 192.168.0.2.65526 > 192.168.0.20.10498: [bad udp cksum 3100!]
udp 24 (ttl 64, id 930)
0x0000   4500 0034 03a2 0000 4011 f5b0 c0a8 0002 E..4....@.......
0x0010   c0a8 0014 fff6 2902 0020 556d 7374 7265 ......)...Umstre
0x0020   616d 2f31 3932 2e31 3638 2e30 2e31 3030 am/192.168.0.100
0x0030   2f31 300a                                      /10.

01:41:23.674796 192.168.0.2.65525 > 192.168.0.20.10498: [bad udp cksum 4a00!]
udp 49 (ttl 64, id 1031)
0x0000   4500 004d 0407 0000 4011 f532 c0a8 0002 E..M....@..2....
0x0010   c0a8 0014 fff5 2902 0039 a9b4 6d73 7472 ......)..9..mstr
0x0020   6561 6d2f 3139 322e 3136 382e 302e 313a eam/192.168.0.1:
0x0030   3139 322e 3136 382e 302e 3130 303a 3139 192.168.0.100:19
0x0040   322e 3136 382e 302e 322f 3130 0a        2.168.0.2/10.

01:41:23.675771 arp who-has 192.168.0.1 tell 192.168.0.20
0x0000   0001 0800 0604 0001 0010 5a99 6544 c0a8 ..........Z.eD..
0x0010   0014 0000 0000 0000 c0a8 0001 0000 0000 ................
0x0020   0000 0000 0000 0000 0000 0000 0000      ..............

01:41:23.675772 arp who-has 192.168.0.100 tell 192.168.0.20
0x0000   0001 0800 0604 0001 0010 5a99 6544 c0a8 ..........Z.eD..
0x0010   0014 0000 0000 0000 c0a8 0064 0000 0000 ...........d....
0x0020   0000 0000 0000 0000 0000 0000 0000      ..............

01:41:23.675773 77.172.43.85.38444 > 192.168.0.2.26296: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 50237)
0x0000   4508 0028 c43d 0000 ff06 bdde 4dac 2b55 E..(.=......M.+U
0x0010   c0a8 0002 962c 66b8 ea97 d237 0000 0000 .....,f....7....
0x0020   5010 4000 7c74 0000 0000 0000 0000      P.@.|t........

01:41:23.675774 88.148.222.45.39212 > 192.168.0.2.10342: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 51005)
0x0000   4508 0028 c73d 0000 ff06 fd1d 5894 de2d E..(.=......X..-
0x0010   c0a8 0002 992c 2866 ed97 d237 0000 0000 .....,(f...7....
0x0020   5010 4000 f705 0000 0000 0000 0000      P.@...........

01:41:23.675775 0.18.219.113.39980 > 192.168.0.2.41622: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 51773)
0x0000   4508 0028 ca3d 0000 ff06 555c 0012 db71 E..(.=....U\...q
0x0010   c0a8 0002 9c2c a296 f097 d237 0000 0000 .....,.....7....
0x0020   5010 4000 d213 0000 0000 0000 0000      P.@...........

01:41:23.675776 121.161.140.109.40748 > 192.168.0.2.16749: .  [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 52541)
0x0000   4508 0028 cd3d 0000 ff06 27d1 79a1 8c6d E..(.=....'.y..m
0x0010   c0a8 0002 9f2c 416d f397 d237 0000 0000 .....,Am...7....
0x0020   5010 4000 02b2 0000 0000 0000 0000      P.@...........

01:41:23.675777 79.238.213.72.41516 > 192.168.0.2.46276: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 53309)
0x0000   4508 0028 d03d 0000 ff06 05a9 4fee d548 E..(.=......O..H
0x0010   c0a8 0002 a22c b4c4 f697 d237 0000 0000 .....,.....7....
0x0020   5010 4000 6a32 0000 0000 0000 0000      P.@.j2........

01:41:23.675778 104.24.203.64.42284 > 192.168.0.2.61623: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 54077)
0x0000   4508 0028 d33d 0000 ff06 f486 6818 cb40 E..(.=......h..@
0x0010   c0a8 0002 a52c f0b7 f997 d237 0000 0000 .....,.....7....
0x0020   5010 4000 1a1d 0000 0000 0000 0000      P.@...........

01:41:23.675779 37.60.73.50.43052 > 192.168.0.2.51311: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 54845)
0x0000   4508 0028 d63d 0000 ff06 b671 253c 4932 E..(.=.....q%<I2
0x0010   c0a8 0002 a82c c86f fc97 d237 0000 0000 .....,.o...7....
0x0020   5010 4000 0150 0000 0000 0000 0000      P.@..P........

01:41:23.675780 142.14.73.40.43820 > 192.168.0.2.8979: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 55613)
0x0000   4508 0028 d93d 0000 ff06 4aa9 8e0e 4928 E..(.=....J...I(
0x0010   c0a8 0002 ab2c 2313 ff97 d237 0000 0000 .....,#....7....
0x0020   5010 4000 37e4 0000 0000 0000 0000      P.@.7.........

01:41:23.676748 144.19.212.69.44588 > 192.168.0.2.51668: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 56381)
0x0000   4508 0028 dc3d 0000 ff06 ba86 9013 d445 E..(.=.........E
0x0010   c0a8 0002 ae2c c9d4 0298 d237 0000 0000 .....,.....7....
0x0020   5010 4000 fdff 0000 0000 0000 0000      P.@...........

01:41:23.676749 155.176.45.2.45356 > 192.168.0.2.32793: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 57149)
0x0000   4508 0028 df3d 0000 ff06 532d 9bb0 2d02 E..(.=....S-..-.
0x0010   c0a8 0002 b12c 8019 0598 d237 0000 0000 .....,.....7....
0x0020   5010 4000 dd61 0000 0000 0000 0000      P.@..a........

01:41:23.676750 10.98.211.13.46124 > 192.168.0.2.1995: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 57917)
0x0000   4508 0028 e23d 0000 ff06 3b70 0a62 d30d E..(.=....;p.b..
0x0010   c0a8 0002 b42c 07cb 0898 d237 0000 0000 .....,.....7....
0x0020   5010 4000 3af3 0000 0000 0000 0000      P.@.:.........

01:41:23.676751 214.235.187.89.46892 > 192.168.0.2.14172: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 58685)
0x0000   4508 0028 e53d 0000 ff06 839a d6eb bb59 E..(.=.........Y
0x0010   c0a8 0002 b72c 375c 0b98 d237 0000 0000 .....,7\...7....
0x0020   5010 4000 508c 0000 0000 0000 0000      P.@.P.........

01:41:23.676752 90.193.127.8.47660 > 192.168.0.2.64812: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 59453)
0x0000   4508 0028 e83d 0000 ff06 3916 5ac1 7f08 E..(.=....9.Z...
0x0010   c0a8 0002 ba2c fd2c 0e98 d237 0000 0000 .....,.,...7....
0x0020   5010 4000 3d37 0000 0000 0000 0000      P.@.=7........

01:41:23.676753 160.176.42.60.48428 > 192.168.0.2.17432: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 60221)
0x0000   4508 0028 eb3d 0000 ff06 44f3 a0b0 2a3c E..(.=....D...*<
0x0010   c0a8 0002 bd2c 4418 1198 d237 0000 0000 .....,D....7....
0x0020   5010 4000 ff28 0000 0000 0000 0000      P.@..(........
----------------------------------------------------------------------------

An attack (only packets including "0" octets are shown) would
similarly be seen with Cisco Net Flows like this:

------------------------------------------------------------------------------
% grep "[ \.]0[ \.(]" ddos-000415
Apr 15 04:12:08 tcp 82.0.151.5(29497) -> 192.168.10.5(27072), 1 packet
Apr 15 04:12:18 tcp 207.0.149.32(21893) -> 192.168.10.5(3913), 1 packet
Apr 15 04:12:33 tcp 0.147.151.82(10473) -> 10.4.152.237(2810), 1 packet
Apr 15 04:13:39 tcp 60.0.33.36(41079) -> 10.4.152.237(31754), 1 packet
Apr 15 04:14:03 tcp 103.140.148.0(4247) -> 10.4.152.237(29689), 1 packet
Apr 15 04:14:15 tcp 214.1.99.0(46714) -> 10.4.152.237(22524), 1 packet
Apr 15 04:15:11 tcp 10.148.60.0(12276) -> 192.168.10.5(31122), 1 packet
Apr 15 04:15:20 tcp 0.112.67.108(4550) -> 192.168.10.5(63787), 1 packet
Apr 15 04:15:33 tcp 13.0.16.2(39092) -> 10.4.152.237(57998), 1 packet
 . . .  Apr 15 06:45:24 tcp 18.167.171.0(54104) -> 10.200.5.8(32779),
1 packet Apr 15 06:45:52 tcp 0.23.15.38(45621) -> 10.200.5.8(20780),
1 packet Apr 15 06:46:14 tcp 0.12.109.77(38670) -> 10.200.5.8(47776),
1 packet Apr 15 07:19:12 tcp 199.120.0.72(64912) ->
10.4.152.237(45151), 1 packet Apr 15 07:27:37 tcp 0.28.232.21(52533)
-> 10.4.152.237(338), 1 packet Apr 15 07:28:13 tcp 99.61.233.0(20951)
-> 10.4.152.237(58427), 1 packet Apr 15 07:31:23 tcp
195.0.3.111(17193) -> 10.4.152.237(14601), 1 packet Apr 15 07:32:19
tcp 61.108.245.0(24309) -> 10.4.152.237(32809), 1 packet
------------------------------------------------------------------------------

It should also be noted that some of the forged source addresses are
broadcast addresses, multicast addresses, or network addresses,
which can have ramifications if packets are directed back to the
flooding systems (see [12]).

Analysis of the de-compiled agent source code shows the stream2.c
attack is altered slightly to randomize more header fields, with some
static values that can be noted when analyzing network traffic:

------------------------------------------------------------------------------
  packet.ip.ip_id = rand();
  . . .
  packet.tcp.th_win = htons(16384);
  . . .
  packet.tcp.th_seq = random();
  . . .
  packet.tcp.th_sport = rand();
  packet.tcp.th_dport = rand();
  . . .
  while (time(0) <= endtime) {
    if (floodtype != 0) {
      i = 0;
      while (arg4[i] != NULL) { /* until list exhausted */
        if (strchr(arg4[i],'.') != NULL) { /* valid ip */
          packet.ip.ip_dst.s_addr = inet_addr(arg4[i]);
          cksum.pseudo.daddr = inet_addr(arg4[i]);
          s_sin.sin_addr.s_addr = inet_addr(arg4[i]);

          cksum.pseudo.saddr = packet.ip.ip_src.s_addr = random();
          packet.ip.ip_id++;
          packet.tcp.th_sport++;
          packet.tcp.th_seq++;

          s_in.sin_port = packet.tcp.th_dport = rand();
  . . .
         }
      }
    }
  }
------------------------------------------------------------------------------


Defenses
--------

There are none.  We are all doomed.  Time to shop for property in
Montana and stock up on non-perishable food items and semi-automatic
weapons.

(Seriously, we didn't have time to finish this section.  Any defenses
already discussed for stream/stream2 [12] or other DDoS tools [08]
would apply.  And for God's sake, SECURE ALL THOSE SYSTEMS they are
using to build these networks! ;)


Weaknesses
----------

Control communication (attacker <-> handler and handler <-> agent)
is not encrypted, and is thus subject to session hijacking and
third-party control, respectively.  Agents do not authenticate the
source of commands, so you can easily use the "ping"/"pong" feature to
detect agents that are not currently streaming.

In fact, "pong" is the only response sent back to the handler, and
"newserver" is only sent to handler(s) when the program first starts.
(The installation of the agent described in Appendix D causes it to
be run after each boot.  This would mean a set of "newserver" packets
could be detected at this point, although it is not recommended that
rebooting be used as the first step in verifying an agent
installation.  It is better to follow the more thorough forensic
analysis techniques detailed in Appendices D and E to ensure as
little evidence as possible is deleted or altered.)

The agent will segmentation fault upon receiving a badly formatted
command, e.g.  "stream foo bar". It is conceivable that such
fault generation can disable agents, but they will most likely get
restarted at the next reboot of the host.

Flooding an agent with requests will saturate its file handle limit
and cause it to become unresponsive. Since no notification or
acknowledgement is made to the handler, one can easily send >1024
commands to the agent. Most floods are short, so it may be possible to
"squeeze in there".

The agent process is single-threaded, implying that it will not
process incoming commands while in mid-flood.  This renders any ideas
about remotely crashing or halting a flooding agent unworkable.

The agent also will, in multi-flood mode, flood all hosts in the
colon-delimited list equally for the same amount of time.  This seems
to either implicate packet loss due to congestion or require that
multiple flooding machines with different flood durations were in play
to explain evidence witnessed in argus flows at third party sites,
showing the side-effects of multiple targets being hit at the same
time with spoofed IPs (namely ICMP "Host Unreachable" packets and TCP
RST packets).

One thing that has been witnessed is that when there are multiple
targets which start at the same time there is a tailing-off with some
targets being hit longer than others and the rate of spoofed packets
also seems to decline with time - sort of dwindling.  (We ran out
of time to analyze third-party effects of flooding, but there will
no doubt be followup on this topic on BUGTRAQ.)


The next logical evolutionary steps
-----------------------------------

It is not a stretch to assume that features from other published DDoS
tools will make their way into tools like mstream.  Briefly, some
likely enhancements to this code include:

1) Adding source port filtering for connections to handler.
2) Adding authentication between handler and agent.
3) Packet size selection.
4) Flags for flood packets (ACK, RST, NUL, random, whatever).
5) Encryption for attacker<->handler traffic.
6) More obfuscation of embedded commands.

FIN

Credits
-------

As principal author of this particular analysis, I guess I get to
do the shouts, greetz, and mad props.

Since last fall, I've spent every major holiday analyzing a DDoS tool.
Halloween (trinoo), Thanksgiving (TFN), Christmas (stacheldraht) and
New Years (building/testing scanning tools for all three).  Sven
Dietrich (no, we are not related and are not the same person as some
have mistaken! ;) took the lead on analyzing "shaft" [07] along with
Neil Long, allowing me to actually relax a bit on President's and
Valentine's days.

Then some DDoS attacks brought mstream out of the shadows, and over
the Easter Holiday weekend George Weaver kicked butt hand-decompiling
the agent, Sven Dietrich obtaining packet and system call traces on a
test network, and Neil Long gathering third-party effects data to
correlate with actual attacks, while I analyzed forensic data
gathered from an agent system and tried to keep up with the rest.
Its amazing what can get done when four people consume that much
sweets and hard boiled eggs.

Credit also goes to Andrew Korty and investigators at Indiana
University for their forensic analysis and data gathering.  Site to
site cooperation and coordination is key to incident response in DDoS
attacks, and the more people who know about this and learn the
techniques, the better off we'll all be.

I should also thank Anonymous, who posted the source code for mtsream
through Security Focus [20], acknowledging me by name for my earlier
work.  My only reservation is that the result of publishing the
source (instead of sending it to me directly) was to alter all of our
weekend plans to rush this analysis out (e.g., I was planning on
doing a training hike all day Sunday.) Whoever you are, I should
think you owe me at least a sixer of good microbrew for that! ;)
(The upside is that this simplifies the problem of getting code into
the hands of all vendors, incident response teams, and security
software authors in a fair and above board way.)

--
Dave Dittrich <dittrich@cac.washington.edu>
University of Washington



Appendix A - References
-----------------------

[00] TCP/IP Illustrated, Vol. I, II, and III. W. Richard Stevens and Gary
    R. Wright., Addison-Wesley

[01] CERT Distributed System Intruder Tools Workshop report
	http://www.cert.org/reports/dsit_workshop.pdf

[02] CERT Advisory CA-99-17 Denial-of-Service Tools
	http://www.cert.org/advisories/CA-99-17-denial-of-service-tools.html

[03] The DoS Project's "trinoo" distributed denial of service attack
    tool, David Dittrich
	http://staff.washington.edu/dittrich/misc/trinoo.analysis

[04] The "Tribe Flood Network" distributed denial of service attack
    tool, David Dittrich
	http://staff.washington.edu/dittrich/misc/tfn.analysis

[05] The "stacheldraht" distributed denial of service attack tool,
     David Dittrich
	http://staff.washington.edu/dittrich/misc/stacheldraht.analysis

[06] TFN2K - An Analysis, Jason Barlow and Woody Thrower, Axent
     Security Team
	http://packetstorm.securify.com/distributed/TFN2k_Analysis-1.3.txt

[07] An analysis of the ``Shaft'' distributed denial of service tool,
     Sven Dietrich, Neil Long, and David Dittrich
	http://netsec.gsfc.nasa.gov/~spock/shaft_analysis.txt

[08] Distributed Denial of Service (DDoS) Attack Tools, David Dittrich
	http://staff.washington.edu/dittrich/misc/ddos/

[09] Distributed denial of service attack tools at Packet Storm Security
	http://packetstorm.securify.com/distributed/

[10] "Root Kits" and hiding files/directories/processes after a break-in,
     David Dittrich
	http://staff.washington.edu/dittrich/misc/faqs/rootkits.faq

[11] Technical details of the attack on Yahoo!
	http://packetstorm.securify.com/distributed/yahoo.txt

[12] BUGTRAQ threads on the stream.c DoS attack and its fallout
	http://staff.washington.edu/dittrich/misc/ddos/stream.txt

[13] RFC 2267 -- Network Ingress Filtering: Defeating Denial of Service
     Attacks which employ IP Source Address Spoofing, Paul Fergussen
     and Daniel Senie
	ftp://ftp.isi.edu/in-notes/rfc2267.txt

[14] ngrep
	http://www.packetfactory.net/ngrep/

[15] rid
	http://theorygroup.com/Software/RID

[16] Dan Farmer & Wietse Venema's class on computer forensic analysis
	http://www.fish.com/security/forensics.html

[17] tcpdump
	ftp://ftp.ee.lbl.gov/tcpdump.tar.Z

[18] snort
	http://www.clark.net/~roesch/security.html

[19] netcat ("nc"), Hobbit
	http://packetstorm.securify.com/UNIX/netcat/nc110.tgz

[20] Source code for mstream
	http://securityfocus.com/templates/archive.pike?list=82&date=2000-04-29&thread=200004291748.TAA13203@lobeda.jena.thur.de


Appendix B - Example snort rules for detecting mstream
------------------------------------------------------

alert UDP any any -> any 6838 (msg:
"IDS100/ddos-mstream-agent-to-handler"; content: "newserver"; ) alert
UDP any any -> any 10498 (msg:
"IDS101/ddos-mstream-handler-to-agent"; content: "stream/"; ) alert
UDP any any -> any 10498 (msg:
"IDS102/ddos-mstream-handler-ping-to-agent" ; content: "ping";) alert
UDP any any -> any 10498 (msg:
"IDS103/ddos-mstream-agent-pong-to-handler" ; content: "pong";) alert
TCP any any -> any 12754 (msg:
"IDS109/ddos-mstream-client-to-handler"; flags: S;) alert TCP any
12754 -> any any (msg: "IDS110/ddos-mstream-handler-to-client";
content: ">"; flags: AP;) alert TCP any any -> any 15104 (msg:
"IDS111/ddos-mstream-client-to-handler"; flags: S;) alert TCP any
15104 -> any any (msg: "IDS112/ddos-mstream-handler-to-client";
content: ">"; flags: AP;)



Appendix C - Rid templates for detecting mstream
------------------------------------------------

start mstream-wild
        send udp dport=10498 data="ping"
        recv udp dport=6838 data="pong" nmatch=2
end mstream-wild
start mstream-published
        send udp dport=7983 data="ping"
        recv udp dport=9325 data="pong" nmatch=2
end mstream-published


Appendix D - Initial Intrusion and Concealment on Agent system
--------------------------------------------------------------

Examination of an agent system, and interviewing the owner, identified
what looks like possibly two separate compromises.  One sometime
before March 31, 2000 (a password file entry for "inertia" existed
until removed by the system owner on April 1), the transfer and
installation of a rootkit (lrk4) and DDoS agent ("rpc.wall") on April
13 16:02 (all times, unless otherwise noted, are US/Pacific, or
GMT-0700), and the marks of an ADM named attack and login on April 15
05:55.  System logs had been deleted and/or scrubbed, so log evidence
was not useful in determining what occurred on the system.

Some of these activities can still be identified using the "mactime"
program, part of Dan Farmer and Wietse Venema's "Coroner's Toolkit"
[16]. [The date "100" is, obviously, a Perl Y2K bug in the "mactime"
program.  This software was graciously provided by Dan Farmer, even
though it is not publicly available at this time.  It definitely
shows how useful Unix forensic analysis tools can be.]

------------------------------------------------------------------------------
Apr 13 100 16:02:42  12060 .aa -rwxr-xr-x root/www root     /bin/chown
                     12660 m.m -r-sr-xr-x root/www bin      /bin/login
Apr 13 100 16:02:43  2048 mcmc drwxr-xr-x root/www root     /bin
                     12660  cc -r-sr-xr-x root/www bin      /bin/login
                    168748 .a. -rwxr-xr-x root/www root     /usr/bin/as
                     64796 .a. -rwxr-xr-x root/www root     /usr/bin/egcs
                     64796 .a. -rwxr-xr-x root/www root     /usr/bin/gcc
                     64796 .a. -rwxr-xr-x root/www root     /usr/bin/i386-redhat-linux-gcc
                    168496 .a. -rwxr-xr-x root/www root     /usr/bin/ld
                     12656 m.c -rws--x--x root/www root     /usr/bin/old
                     12656 m.c -r-xr-xr-x root/www bin      /usr/bin/xstat
                      2315 .a. -rw-r--r-- root/www root     /usr/include/_G_config.h
                      1313 .a. -rw-r--r-- root/www root     /usr/include/alloca.h
                      4090 .a. -rw-r--r-- root/www root     /usr/include/arpa/inet.h
                      3451 .a. -rw-r--r-- root/www root     /usr/include/bits/byteswap.h
                     13327 .a. -rw-r--r-- root/www root     /usr/include/bits/confname.h
                       168 .a. -rw-r--r-- root/www root     /usr/include/bits/endian.h
                      2283 .a. -rw-r--r-- root/www root     /usr/include/bits/errno.h
                      5107 .a. -rw-r--r-- root/www root     /usr/include/bits/fcntl.h
                      4647 .a. -rw-r--r-- root/www root     /usr/include/bits/in.h
                      3406 .a. -rw-r--r-- root/www root     /usr/include/bits/posix_opt.h
                      2842 .a. -rw-r--r-- root/www root     /usr/include/bits/select.h
                      4673 .a. -rw-r--r-- root/www root     /usr/include/bits/sigset.h
                      1716 .a. -rw-r--r-- root/www root     /usr/include/bits/sockaddr.h
                      9033 .a. -rw-r--r-- root/www root     /usr/include/bits/socket.h
                      1297 .a. -rw-r--r-- root/www root     /usr/include/bits/stdio_lim.h
                      2015 .a. -rw-r--r-- root/www root     /usr/include/bits/time.h
                      4673 .a. -rw-r--r-- root/www root     /usr/include/bits/types.h
                      1781 .a. -rw-r--r-- root/www root     /usr/include/bits/uio.h
                      1798 .a. -rw-r--r-- root/www root     /usr/include/endian.h
                      2481 .a. -rw-r--r-- root/www root     /usr/include/errno.h
                      4579 .a. -rw-r--r-- root/www root     /usr/include/fcntl.h
                      9433 .a. -rw-r--r-- root/www root     /usr/include/features.h
                      5861 .a. -rw-r--r-- root/www root     /usr/include/getopt.h
                       973 .a. -rw-r--r-- root/www root     /usr/include/gnu/stubs.h
                     10291 .a. -rw-r--r-- root/www root     /usr/include/libio.h
                     17327 .a. -rw-r--r-- root/www root     /usr/include/netdb.h
                     10779 .a. -rw-r--r-- root/www root     /usr/include/netinet/in.h
                      1591 .a. -rw-r--r-- root/www root     /usr/include/netinet/in_systm.h
                      9086 .a. -rw-r--r-- root/www root     /usr/include/netinet/ip.h
                      4855 .a. -rw-r--r-- root/www root     /usr/include/netinet/tcp.h
                      2550 .a. -rw-r--r-- root/www root     /usr/include/rpc/netdb.h
                      6467 .a. -rw-r--r-- root/www root     /usr/include/stdint.h
                     20816 .a. -rw-r--r-- root/www root     /usr/include/stdio.h
                     27654 .a. -rw-r--r-- root/www root     /usr/include/stdlib.h
                     13245 .a. -rw-r--r-- root/www root     /usr/include/string.h
                      2104 .a. -rw-r--r-- root/www root     /usr/include/strings.h
                      4932 .a. -rw-r--r-- root/www root     /usr/include/sys/cdefs.h
                      3359 .a. -rw-r--r-- root/www root     /usr/include/sys/select.h
                      7996 .a. -rw-r--r-- root/www root     /usr/include/sys/socket.h
                      1577 .a. -rw-r--r-- root/www root     /usr/include/sys/sysmacros.h
                      5337 .a. -rw-r--r-- root/www root     /usr/include/sys/time.h
                      5299 .a. -rw-r--r-- root/www root     /usr/include/sys/types.h
                      1907 .a. -rw-r--r-- root/www root     /usr/include/sys/uio.h
                      9314 .a. -rw-r--r-- root/www root     /usr/include/time.h
                     36708 .a. -rw-r--r-- root/www root     /usr/include/unistd.h
                       874 .a. -rw-r--r-- root/www root     /usr/lib/crtn.o
                   1446620 .a. -rwxr-xr-x root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/cc1
                     46816 .a. -rwxr-xr-x root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/collect2
                     88444 .a. -rwxr-xr-x root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/cpp
                      1424 .a. -rw-r--r-- root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/crtend.o
                      5794 .a. -rw-r--r-- root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/include/stdarg.h
                      9834 .a. -rw-r--r-- root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/include/stddef.h
                    770000 .a. -rw-r--r-- root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/libgcc.a
                      1957 .a. -rw-r--r-- root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/specs
                       178 .a. -rw-r--r-- root/www root     /usr/lib/libc.so
                     69638 .a. -rw-r--r-- root/www root     /usr/lib/libc_nonshared.a
                      6162 .a. -rw-r--r-- 1046     squid    /usr/src/linux/include/asm-i386/errno.h
                      1492 .a. -rw-r--r-- 1046     squid    /usr/src/linux/include/asm-i386/socket.h
                       277 .a. -rw-r--r-- 1046     squid    /usr/src/linux/include/asm-i386/sockios.h
                       305 .a. -rw-r--r-- 1046     squid    /usr/src/linux/include/linux/errno.h
Apr 13 100 16:02:44    702 mcmc -rwxr-xr-x root/www root     /etc/rc.d/rc.local
                      1024 mcmc drwxr-xr-x root/www root     /root/.ncftp
                         9 mcmc lrwxrwxrwx root/www root     /root/.ncftp/history
                         9 mcmc lrwxrwxrwx root/www root     /root/.ncftp/log
                         9 mcmc lrwxrwxrwx root/www root     /root/.ncftp/trace
                     29696 m.c drwxr-xr-x root/www root     /usr/bin
                     17016 m.c -rwxr-xr-x root/www root     /usr/bin/rpc.wall
                      8460 .a. -rw-r--r-- root/www root     /usr/lib/crt1.o
                      1124 .a. -rw-r--r-- root/www root     /usr/lib/crti.o
                      1892 .a. -rw-r--r-- root/www root     /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/crtbegin.o
 . . .
Apr 15 100 05:55:09    1024 mcmc drwxr-xr-x root/www root     /var/named
                      1024 mcmc drwxr-xr-x root/www root     /var/named/ADMROCKS
Apr 15 100 05:56:19   20437 .a. -rwxr-xr-x root/www root     /usr/sbin/tcpd
Apr 15 100 05:56:20      34 .aa -rw-r--r-- root/www root     /usr/libexec/awk/addy.awk
                     35628 .a. -rwxr-xr-x root/www root     /usr/sbin/in.telnetd
Apr 15 100 05:56:26  159576 .a. -rwxr-xr-x root/www root     /usr/bin/pico
                       975 .a. -rw-r--r-- root/www root     /usr/share/terminfo/v/vt200
                       975 .a. -rw-r--r-- root/www root     /usr/share/terminfo/v/vt220
------------------------------------------------------------------------------

From this listing, the following observations can be made:

o On April 13 at 16:02, /bin/login was created and /bin/chown run.

o At the same time, the compile was run (gcc/egcs) and /bin/old and
  /bin/xstat created.  Access times on the .h files listed show the
  program being compiled uses network libraries.

o Next, the /etc/rc.d/rc.local file is modified to include the
  line "/usr/bin/rpc.wall" at the end, thus re-starting the agent on
  each reboot. ncftp logging files are modified (they were deleted
  and turned into links to /dev/null to disable logging of ncftp
  file transfers.)

o The program /usr/bin/rpc.wall was modified, and C runtime libraries
  accessed, which implies it was run.  (This cannot be confirmed
  because the rootkit prevented the program from being seen by
  the administrator, who also rebooted the system and restarted it,
  thus modifying the /usr/bin/rpc.wall access date.)

o On April 15 at 05:55, it appears that the ADM named buffer overrun
  exploit was used against this system, followed within the minute
  access to a tcpd wrapped service which invoked in.telnetd.
  The rootkit configuration file /usr/libexec/awk/addy.awk was
  also accessed.  [It is not clear if this was a just a coincidental
  second (third?) intrusion attempt.]

o Six seconds later /usr/bin/pico was run and the vt200 terminfo
  definition was accessed.  Since the trojan version of login contains
  the string "vt200", this confirms the backdoor that was installed
  two days earlier was used to gain root access.

While the /etc/passwd entry for the "inertia" account had been deleted
by the administrator of the system, the /etc/shadow entry (original
modification date not known) was not:

------------------------------------------------------------------------------
inertia:iUCNir1cd8pI2:::::::
------------------------------------------------------------------------------

The password, "hi", was cracked in a fraction of a second.

Strings in "/bin/login" show the classic trojan horse signs of a path
to a non-standard program ("/usr/bin/xstat") and embedded terminal
type ("vt200") that triggers a root shell:

------------------------------------------------------------------------------
 . . .
login
/bin/sh
/usr/bin/xstat
TERM
bcshjvmudzwxftejk
vt200
%s=%s
init.c
 . . .
------------------------------------------------------------------------------

Strings in "/bin/ls" and "/bin/ps" show the names of the rootkit
configuration files to be "/usr/libexec/awk/files.awk" and
"/usr/libexec/awk/ps.awk" (respectively).  Another configuration file
is seen above, "/usr/libexec/awk/addy.awk".

The big tipoff that this is a standard Linux rootkit is the compiler
inserted debugging information that includes the (edited) path to the
source file, which had better not be a valid path on Red Hat's
development system:

------------------------------------------------------------------------------
 . . .
ls.c
/home/XXXXX/stuff/lrk4/fileutils-3.13/src/
gcc2_compiled.
int:t1=r1;-2147483648;2147483647;
char:t2=r2;0;127;
 . . .
------------------------------------------------------------------------------

Since telnetd was not otherwise used on the system (instead SSH was
used for remote login), the intruder placed an entry (date unknown)
in /etc/inetd.conf to run in.telnetd as service "working":

------------------------------------------------------------------------------
 . . .
#finger	stream	tcp	nowait	root	/usr/sbin/tcpd	in.fingerd
#cfinger stream	tcp	nowait	root	/usr/sbin/tcpd	in.cfingerd
#systat	stream	tcp	nowait	guest	/usr/sbin/tcpd	/bin/ps	-auwwx
#netstat	stream	tcp	nowait	guest	/usr/sbin/tcpd	/bin/netstat	-f inet
working	stream	tcp	nowait	root	/usr/sbin/tcpd	in.telnetd
 . . .
------------------------------------------------------------------------------

The following entry was added (date unknown) to the /etc/services file
for this service:

------------------------------------------------------------------------------
working		1120/tcp			# Kerberos working daemon
------------------------------------------------------------------------------

This service port is shown in (edited) "lsof" output as listening:

------------------------------------------------------------------------------
inetd      353     root    4u  IPv4        375               UDP *:talk
inetd      353     root    5u  IPv4        376               UDP *:ntalk
inetd      353     root    6u  IPv4        377               TCP *:working (LISTEN)
inetd      353     root    8u  IPv4        378               TCP *:time (LISTEN)
inetd      353     root   10u  IPv4        379               UDP *:time
inetd      353     root   11u  IPv4        380               TCP *:auth (LISTEN)
inetd      353     root   12u  IPv4        381               TCP *:linuxconf (LISTEN)
------------------------------------------------------------------------------

It was noted earlier that the trojan version of /bin/login allows
access using a terminal type of "vt200" (as seen by strings in the
binary), and that the mactime output shows the termcap entries for
vt200 are accessed right after the backdoor in.telnetd was accessed.
This places the last login to the system using this backdoor at
approximately 05:56 on April 15 (per the system's clock).  This is
within the time window of one of the attacks logged by the network
operations engineers (Apr 15 04:11:35 to 07:32:22).


Appendix E - Analysis of handler system from Indiana University
---------------------------------------------------------------

[Note - this, too, was edited prior to publication.]


On 20 April 2000 the IT Security Office at Indiana University was
contacted by Dave Dittrich of the University of Washington (and
subsequently by Penn State).  A system had been identified attempting
to use an agent similar to Trinoo to initiate denial of service
attacks using spoofed source addresses.  The agent binary contained
an IP address for a host on one of IU's networks.

Our first step was to have our networks group block all traffic to
and from this IP address and log any inbound attempts.  The main
intention was to prevent the machine from causing more trouble,
but we also wanted to keep the intruder from closing up shop once
our presence was known.  We avoided rebooting the machine and making
physical changes to the network (e.g., unplugging the LAN cable)
in case the software running on the machine was able to detect such
events and delete itself.

Once the system was isolated from the rest of the network, we ran
the lsof program, the output of which can be found at the end of
this report.

This output indicates a process rpc.wall listening on several UDP
ports, including 6838, to which Mr. Dittrich had noted his agent
process had been sending packets.  The /usr/bin/... has been opened
for reading hundreds of times.  This file contained a list of IP
addresses, enciphered by a simple ASCII rotation.  These IP addresses
seem to be those of agent servers controlled by this master.  Indeed,
the identified agent system was on the list.  We found a similar
file, /dev/grab/..., which we think might have been used by another
master server.

An example line from one of these files might look like

	ckd`chj`cc`jb<

The cipher merely adds 50 to the ASCII value of each character, so the
resulting IP is

	192.168.11.80

Such a file can be translated by passing it through the UNIX command

	tr 'b-k`' '0-9.' | sed 's/<$//'

Using this translation, we found 76 IP addresses on the system,
presumably all DDoS slave agents.

Source code for the rpc.wall program was found by searching through
the raw disk.  The attacker had done a good job of deleting the
original source file but failed to use the -pipe option to gcc.
Preprocessed code was therefore written to a temporary file to be
passed to the compiler proper.  Though this temporary file was
deleted from the filesystem by the compiler, the data itself remained
intact on the disk.

Analyzing the source code of this program, we found it can perform
"stream" as well as "mstream" attacks.  An mstream attack is
apparently a simultaneous stream attack on multiple IPs.  We believe
this added functionality could result in larger-scale DDoS attacks
than we have seen before.

The source code found is included at the end of this report.

At this point we felt we had most of the important evidence, so we
used the dd program to copy the entire filesystem over the network
to my notebook.  From there we could mount the file readonly and
examine it later.  Here's what we have found so far:

ROOT KIT

A root kit called flukek.tgz was found in /bin.  It contains
replacements for, among many others, cron, in.timed, inetd, login,
named, passwd, rshd, syslogd, tcpd.  The following files were added
to /bin, /sbin, /usr/bin, and /usr/sbin since the system was
installed.  Nothing appeared to have been added to /usr/libexec or
anything under /usr/local.  Times listed are inode change times.  The
file /usr/bin/old appears to be the old login binary.

-rw-r--r--  1 root  wheel   2387704 Dec 18 16:37 bin/flukek.tgz
-r-sr-xr-x  1 root  daemon    12656 Apr 13 23:39 bin/login
-rw-r--r--  1 root  wheel      1401 Apr 24 12:11 usr/bin/...
-rwsr-xr-x  1 root  wheel     20164 Mar 31 14:50 usr/bin/old
-rwxr-xr-x  1 root  wheel     33610 Apr 13 23:37 usr/bin/rpc.wall
-rwxr-xr-x  1 root  wheel     32727 Apr  5 21:56 usr/bin/xfs
-r-xr-xr-x  1 root  daemon    20164 Mar 31 14:50 usr/bin/xstat
-rwxr-xr-x  1 root  wheel    111500 Mar 31 20:09 usr/sbin/dnskeygen
-rwxr-xr-x  1 root  wheel    266712 Mar 31 20:09 usr/sbin/irpd
-rwxr-xr-x  1 root  wheel    528612 Mar 31 20:09 usr/sbin/named
-rwxr-xr-x  1 root  wheel      7166 Mar 31 20:09 usr/sbin/named-bootconf
-rwxr-xr-x  1 root  wheel    285076 Mar 31 20:09 usr/sbin/named-xfer
-rwxr-xr-x  1 root  wheel     37056 Mar 31 20:09 usr/sbin/ndc

No libraries, kernel modules, or PAM modules appear to have been
replaced.

ACCOUNTS ADDED

The /etc/passwd and /etc/shadow files were recently modified:

-rw-r--r--  1 root  wheel  849 Feb 17 00:57 /mnt/etc/passwd
-rw-------  1 root  wheel  884 Feb 17 00:57 /mnt/etc/passwd-
-r--------  1 root  wheel  794 Feb 17 00:57 /mnt/etc/shadow
-r--------  1 root  wheel  658 Nov 15 10:07 /mnt/etc/shadow-

The backup of /etc/shadow made 15 November gives us a clue what
accounts were added:

	www:MyjKA0KGHplq6:11004:0:99999:7:::
	login1:MyjKA0KGHplq6:11004:0:99999:7:::
	web:af47L/OTL7K6.:11004:0:99999:7:::
	x::11004:0:99999:7:::

I have not yet attempted to crack these passwords.  I did try to
log in as "x" but was denied access.

INETD SERVICE ADDED

Inspection of /etc/services and /etc/inetd.conf reveals a service
called "a", running in.telnetd on TCP port 1111, has been added.

/etc/services:

	a 1111/tcp

/etc/inetd.conf

	a stream tcp nowait root /usr/sbin/tcpd in.telnetd

I haven't attempted to connect to it yet.

LOG FILES

The following three files were the only log files found to have
any pertinent information.

/.bash_history:

nslookup
cd /bin
w
ps x
ftp 192.168.0.1 21
w

/var/log/secure:

Mar 29 18:39:18 herc in.ftpd[824]: connect from 10.156.97.157
Mar 29 19:29:15 herc in.ftpd[876]: connect from 10.156.97.111
Mar 29 19:49:58 herc in.ftpd[882]: connect from 10.156.97.111
Mar 29 19:50:21 herc in.ftpd[887]: connect from 10.156.97.111
Mar 31 14:58:14 herc in.telnetd[4224]: connect from 10.54.115.105
Apr  3 23:54:02 herc in.telnetd[10403]: connect from 10.72.135.165
Apr  4 05:44:34 herc in.telnetd[11235]: connect from 10.103.26.127
Apr  4 08:28:28 herc in.ftpd[11397]: connect from 10.31.68.158
Apr  4 11:36:16 herc in.ftpd[11565]: connect from 10.31.68.158
Apr  7 05:33:32 herc in.telnetd[16737]: connect from 10.22.82.6
Apr  7 07:32:19 herc in.telnetd[16849]: connect from 10.22.82.6
Apr  7 07:33:01 herc in.telnetd[16851]: connect from 10.22.82.6
Apr  7 07:33:20 herc in.ftpd[16852]: connect from 10.22.82.6
Apr  7 07:34:11 herc in.ftpd[16855]: connect from 10.22.82.6
Apr  7 07:35:22 herc in.ftpd[16859]: connect from 10.22.82.6
Apr  7 07:37:02 herc in.rlogind[16860]: connect from 10.22.82.2
Apr  7 07:37:12 herc in.fingerd[16863]: connect from 10.22.82.2
Apr  7 07:37:18 herc in.rexecd[16866]: connect from 10.22.82.2
Apr  7 07:37:22 herc in.rshd[16867]: connect from 10.22.82.2
Apr  7 07:37:24 herc in.telnetd[16868]: connect from 10.22.82.2
Apr  7 07:37:30 herc in.ftpd[16870]: connect from 10.22.82.2
Apr  8 13:53:02 herc in.ftpd[19028]: connect from 10.247.49.53
Apr 10 23:00:05 herc in.ftpd[23304]: connect from 10.8.148.36
Apr 10 23:07:51 herc in.ftpd[23347]: connect from 10.8.148.36
Apr 13 06:50:02 herc in.telnetd[27895]: connect from 10.215.99.125
Apr 13 10:52:27 herc in.ftpd[28170]: connect from 10.114.238.145
Apr 13 10:55:50 herc in.ftpd[28171]: connect from 10.114.238.145
Apr 13 11:02:39 herc in.ftpd[28217]: connect from 10.114.238.145
Apr 16 16:29:47 herc in.ftpd[1734]: connect from 10.161.208.34
Apr 16 16:30:10 herc in.ftpd[1737]: connect from 10.161.208.34
Apr 23 18:59:36 herc in.telnetd[14746]: connect from 10.27.211.234
Apr 24 17:02:03 herc in.telnetd[16505]: connect from 10.79.16.203

/var/log/wtmp (reverse chronological order):

root     pts/2        :0               Mon Apr 24 18:05   still logged in
root     pts/0        :0               Mon Apr 24 17:24   still logged in
ftp      ftp          XXXXXX-XXXXXXXX. Thu Apr 13 10:02 - 10:02 (00:00)
ftp      ftp          XXXXXX-XXXXXXXX. Thu Apr 13 09:55 - 09:56 (00:00)
ftp      ftp          XXXXXXX-X.XXXXXX Mon Apr 10 22:07 - 22:09 (00:01)
ftp      ftp          XXX.XXX.82.6     Fri Apr  7 06:34 - 06:35 (00:00)
ftp      ftp          XXX.XXX.82.6     Fri Apr  7 06:33 - 06:34 (00:00)
ftp      ftp          XXXXX.XX-XXXXXXX Tue Apr  4 10:36 - 10:36 (00:00)
ftp      ftp          XXXXXXXX-XXXX.XX Wed Mar 29 19:50 - 19:50 (00:00)
ftp      ftp          XXXXXXXX-XXXX.XX Wed Mar 29 19:29 - 19:29 (00:00)
reboot   system boot                   Wed Mar 29 16:17 (26+20:09)


OUTPUT OF lsof [edited]

COMMAND     PID USER   FD   TYPE     DEVICE    SIZE       NODE NAME
 . . .
inetd       378 root  cwd    DIR        3,1    1024          2 /
inetd       378 root  rtd    DIR        3,1    1024          2 /
inetd       378 root  txt    REG        3,1   18020     143469 /usr/sbin/inetd
inetd       378 root  mem    REG        3,1  342206      30722 /lib/ld-2.1.1.so
inetd       378 root  mem    REG        3,1 4016683      30729 /lib/libc-2.1.1.so
inetd       378 root  mem    REG        3,1  251436      30766 /lib/libnss_nisplus-2.1.1.so
inetd       378 root  mem    REG        3,1  364235      30742 /lib/libnsl-2.1.1.so
inetd       378 root  mem    REG        3,1  243964      30760 /lib/libnss_files-2.1.1.so
inetd       378 root    0u   CHR        1,3               2071 /dev/null
inetd       378 root    1u   CHR        1,3               2071 /dev/null
inetd       378 root    2u   CHR        1,3               2071 /dev/null
inetd       378 root    4u  inet        506                TCP *:ftp (LISTEN)
inetd       378 root    5u  inet        507                TCP *:telnet (LISTEN)
inetd       378 root    6u  inet        508                TCP *:shell (LISTEN)
inetd       378 root    7r  FIFO        0,0                499 pipe
inetd       378 root    8u  inet        509                TCP *:login (LISTEN)
inetd       378 root    9u   CHR        5,1               2113 /dev/console
inetd       378 root   10u  inet        510                TCP *:exec (LISTEN)
inetd       378 root   11u  inet        511                UDP *:talk
inetd       378 root   12u  inet        512                UDP *:ntalk
inetd       378 root   13u  inet        513                TCP *:finger (LISTEN)
inetd       378 root   14u  inet        514                TCP *:auth (LISTEN)
inetd       378 root   15u  inet        515                TCP *:linuxconf (LISTEN)
inetd       378 root   16u  inet        516                TCP *:a (LISTEN)
inetd       378 root   21w  FIFO        0,0                499 pipe
 . . .
rpc.wall  29108 root  cwd    DIR        3,1   31744      59393 /usr/bin
rpc.wall  29108 root  rtd    DIR        3,1    1024          2 /
rpc.wall  29108 root  txt    REG        3,1   33610      61172 /usr/bin/rpc.wall
rpc.wall  29108 root  mem    REG        3,1  342206      30722 /lib/ld-2.1.1.so
rpc.wall  29108 root  mem    REG        3,1   65996      30758 /lib/libnss_dns-2.1.1.so
rpc.wall  29108 root  mem    REG        3,1 4016683      30729 /lib/libc-2.1.1.so
rpc.wall  29108 root  mem    REG        3,1  243964      30760 /lib/libnss_files-2.1.1.so
rpc.wall  29108 root  mem    REG        3,1  251436      30766 /lib/libnss_nisplus-2.1.1.so
rpc.wall  29108 root  mem    REG        3,1  364235      30742 /lib/libnsl-2.1.1.so
rpc.wall  29108 root  mem    REG        3,1  251787      30764 /lib/libnss_nis-2.1.1.so
rpc.wall  29108 root  mem    REG        3,1  164797      30770 /lib/libresolv-2.1.1.so
rpc.wall  29108 root    0r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root    1u   CHR      136,1                  3 /dev/pts/1
rpc.wall  29108 root    2u   CHR      136,1                  3 /dev/pts/1
rpc.wall  29108 root    3u  inet      36066                TCP *:15104 (LISTEN)
rpc.wall  29108 root    4u   CHR      136,1                  3 /dev/pts/1
rpc.wall  29108 root    5u  inet      36067                UDP *:6838
rpc.wall  29108 root    6r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root    7r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root    8r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root    9r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   10r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   11r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   12r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   13r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   14r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   15r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   16r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   17r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   18r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   19r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   20r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   21u  inet      38990                UDP *:2000
rpc.wall  29108 root   22r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   23u  inet      38992                UDP *:2001
rpc.wall  29108 root   24r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   25u  inet      38996                UDP *:2002
rpc.wall  29108 root   26r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   27u  inet      38997                UDP *:2003
rpc.wall  29108 root   28r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   29u  inet      39001                UDP *:2004
rpc.wall  29108 root   30r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   31u  inet      39002                UDP *:2005
rpc.wall  29108 root   32r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   33u  inet      39003                UDP *:2006
rpc.wall  29108 root   34r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   35u  inet      39007                UDP *:2007
rpc.wall  29108 root   36r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   37u  inet      39011                UDP *:2008
rpc.wall  29108 root   38r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   39u  inet      39012                UDP *:2009
rpc.wall  29108 root   40r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   41u  inet      39016                UDP *:2010
rpc.wall  29108 root   42r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   43u  inet      39017                UDP *:2011
rpc.wall  29108 root   44r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   45u  inet      39018                UDP *:2012
rpc.wall  29108 root   46r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   47u  inet      39019                UDP *:2013
rpc.wall  29108 root   48r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   49u  inet      39020                UDP *:2014
rpc.wall  29108 root   50r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   51u  inet      39027                UDP *:2016
rpc.wall  29108 root   52r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   53u  inet      39028                UDP *:2017
rpc.wall  29108 root   54r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   55u  inet      39029                UDP *:2018
rpc.wall  29108 root   56r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   57u  inet      39033                UDP *:2019
rpc.wall  29108 root   58r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   59u  inet      39034                UDP *:2020
rpc.wall  29108 root   60r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   61u  inet      39038                UDP *:2021
rpc.wall  29108 root   62r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   63u  inet      39039                UDP *:2022
rpc.wall  29108 root   64r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   65r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   66u  inet      39101                UDP *:2024
rpc.wall  29108 root   67r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   68r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   69u  inet      39109                UDP *:2025
rpc.wall  29108 root   70r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   71u  inet      39129                UDP *:2026
rpc.wall  29108 root   72r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   73u  inet      39185                UDP *:2027
rpc.wall  29108 root   74r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   75u  inet      39186                UDP *:2028
rpc.wall  29108 root   76r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   77u  inet      39190                UDP *:2029
rpc.wall  29108 root   78r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   79u  inet      39195                UDP *:2030
rpc.wall  29108 root   80r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   81u  inet      39200                UDP *:2032
rpc.wall  29108 root   82r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   83u  inet      39204                UDP *:2033
rpc.wall  29108 root   84r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   85u  inet      39208                UDP *:2035
rpc.wall  29108 root   86r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   87u  inet      39215                UDP *:2036
rpc.wall  29108 root   88r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   89u  inet      39216                UDP *:2037
rpc.wall  29108 root   90r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   91u  inet      39223                UDP *:2038
rpc.wall  29108 root   92r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   93r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   94r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   95r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   96r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   97r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   98r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root   99r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  100r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  101r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  102r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  103r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  104r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  105r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  106r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  107r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  108r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  109r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  110r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  111r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  112r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  113u  inet      39671                UDP *:2039
rpc.wall  29108 root  114r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  115r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  116r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  117r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  118r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  119r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  120r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  121r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  122r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  123r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  124r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  125r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  126r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  127r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  128r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  129r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  130r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  131r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  132r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  133r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  134r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  135r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  136u  inet      41521                UDP *:2040
rpc.wall  29108 root  137r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  138r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  139r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  140r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  141u  inet      41532                UDP *:2041
rpc.wall  29108 root  142r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  143r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  144u  inet      41614                UDP *:2042
rpc.wall  29108 root  145r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  146u  inet      41615                UDP *:2045
rpc.wall  29108 root  147r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  148u  inet      41622                UDP *:2048
rpc.wall  29108 root  149r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  150u  inet      41681                UDP *:2051
rpc.wall  29108 root  151r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  152u  inet      41685                UDP *:2055
rpc.wall  29108 root  153r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  154u  inet      41690                UDP *:2056
rpc.wall  29108 root  155r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  156u  inet      41694                UDP *:2057
rpc.wall  29108 root  157r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  158u  inet      41696                UDP *:2058
rpc.wall  29108 root  159r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  160u  inet      41706                UDP *:2059
rpc.wall  29108 root  161r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  162u  inet      41707                UDP *:2060
rpc.wall  29108 root  163r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  164u  inet      41708                UDP *:2061
rpc.wall  29108 root  165r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  166u  inet      41715                UDP *:2062
rpc.wall  29108 root  167r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  168u  inet      41716                UDP *:2063
rpc.wall  29108 root  169r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  170r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  171r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  172r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  173r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  174r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  175r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  176r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  177r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  178r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  179r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  180r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  181r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  182r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  183r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  184r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  185r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  186r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  187r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  188r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  189r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  190r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  191r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  192r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  193r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  194r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  195r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  196r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  197r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  198r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  199r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  200r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  201r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  202r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  203r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  204r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  205r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  206r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  207r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  208r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  209r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  210r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  211r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  212r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  213r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  214r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  215r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  216r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  217r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  218r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  219r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  220r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  221r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  222r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  223r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  224r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  225r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  226r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  227r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  228r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  229r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  230r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  231r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  232r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  233r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  234r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  235r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  236r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  237r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  238r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  239r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  240r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  241r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  242r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  243r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  244r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  245r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  246r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  247r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  248r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  249r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  250r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  251r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  252r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  253r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  254r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  255r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  256r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  257r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  258r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  259r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  260r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  261r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  262r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  263r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  264r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  265r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  266r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  267r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  268r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  269r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  270r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  271r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  272r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  273r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  274r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  275r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  276r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  277r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  278r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  279r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  280r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  281r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  282r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  283r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  284u  inet      53028                UDP *:2064
rpc.wall  29108 root  285r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  286u  inet      53029                UDP *:2065
rpc.wall  29108 root  287r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  288u  inet      53039                UDP *:2066
rpc.wall  29108 root  289r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  290u  inet      53040                UDP *:2067
rpc.wall  29108 root  291r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  292u  inet      53048                UDP *:2068
rpc.wall  29108 root  293r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  294u  inet      53052                UDP *:2069
rpc.wall  29108 root  295r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  296r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  297u  inet      53057                UDP *:2070
rpc.wall  29108 root  298r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  299u  inet      53284                UDP *:2071
rpc.wall  29108 root  300r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  301u  inet      53292                UDP *:2072
rpc.wall  29108 root  302r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  303r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  304u  inet      53607                UDP *:2073
rpc.wall  29108 root  305r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  306u  inet      53608                UDP *:2074
rpc.wall  29108 root  307r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  308u  inet      53671                UDP *:2075
rpc.wall  29108 root  309r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  310u  inet      53672                UDP *:2076
rpc.wall  29108 root  311r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  312u  inet      53681                UDP *:2077
rpc.wall  29108 root  313r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  314u  inet      53682                UDP *:2078
rpc.wall  29108 root  315r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  316u  inet      53688                UDP *:2079
rpc.wall  29108 root  317r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  318u  inet      53699                UDP *:2081
rpc.wall  29108 root  319r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  320u  inet      53707                UDP *:2082
rpc.wall  29108 root  321r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  322u  inet      53773                UDP *:2083
rpc.wall  29108 root  323r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  324u  inet      53774                UDP *:2084
rpc.wall  29108 root  325r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  326u  inet      53783                UDP *:2085
rpc.wall  29108 root  327r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  328u  inet      53784                UDP *:2086
rpc.wall  29108 root  329r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  330u  inet      53785                UDP *:2088
rpc.wall  29108 root  331r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  332u  inet      53794                UDP *:2089
rpc.wall  29108 root  333r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  334u  inet      53802                UDP *:2090
rpc.wall  29108 root  335r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  336u  inet      53810                UDP *:2091
rpc.wall  29108 root  337r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  338u  inet      53811                UDP *:2092
rpc.wall  29108 root  339r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  340r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  341u  inet      54862                UDP *:2093
rpc.wall  29108 root  342r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  343r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  344r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  345r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  346r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  347r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  348u  inet      55930                UDP *:2094
rpc.wall  29108 root  349r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  350u  inet      56658                UDP *:2095
rpc.wall  29108 root  351r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  352r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  353r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  354u  inet      57339                UDP *:2096
rpc.wall  29108 root  355r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  356r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  357u  inet      57737                UDP *:2098
rpc.wall  29108 root  358r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  359u  inet      57876                UDP *:2099
rpc.wall  29108 root  360r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  361u  inet      57952                UDP *:2100
rpc.wall  29108 root  362r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  363r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  364r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  365u  inet      58274                UDP *:zephyr-clt
rpc.wall  29108 root  366r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  367u  inet      58855                UDP *:zephyr-hm
rpc.wall  29108 root  368r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  369u  inet      58860                UDP *:2105
rpc.wall  29108 root  370r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  371u  inet      58861                UDP *:2106
rpc.wall  29108 root  372r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  373u  inet      58873                UDP *:2107
rpc.wall  29108 root  374r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  375r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  376r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  377r   REG        3,1    1401     198670 /usr/bin/...
rpc.wall  29108 root  378r   REG        3,1    1401     198670 /usr/bin/...


SOURCE CODE FOR rpc.wall (mstream)

[Source code removed in light of publication.  See [20]]

Andrew J. Korty, Lead Security Engineer
Information Technology Security Office
Office of the Vice President for Information Technology
Indiana University
--
Dave Dittrich                 Client Services
dittrich@cac.washington.edu   Computing & Communications
                              University of Washington

<a href="http://www.washington.edu/People/dad/">
Dave Dittrich / dittrich@cac.washington.edu [PGP Key]</a>

PGP 6.5.1 key fingerprint:
FE 97 0C 57 08 43 F3 EB  49 A1 0C D0 8E 0C D0 BE  C8 38 CC B5
(5054580) ------------------------------------------(Ombruten)

5057051 2000-05-02  21:38  /205 rader/ Postmaster
Mottagare: Bugtraq (import) <10694>
Ärende: Re: Source code to mstream, a DDoS tool
------------------------------------------------------------
Approved-By: aleph1@SECURITYFOCUS.COM
Delivered-To: bugtraq@lists.securityfocus.com
Delivered-To: BUGTRAQ@SECURITYFOCUS.COM
X-Accept-Language: en
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID:  <390E6E79.861F046F@moquijo.com>
Date:         Tue, 2 May 2000 01:58:17 -0400
Reply-To: paul@MOQUIJO.COM
Sender: Bugtraq List <BUGTRAQ@SECURITYFOCUS.COM>
From: Paul Cardon <paul@MOQUIJO.COM>
X-To:         Dave Dittrich <dittrich@CAC.WASHINGTON.EDU>
X-cc:         BUGTRAQ@SECURITYFOCUS.COM, VULN-DEV@SECURITYFOCUS.COM
To: BUGTRAQ@SECURITYFOCUS.COM

Very nice writeup, especially the bits captured in the wild.  I had
almost completed an analysis based on the released source only.
Instead, I will add my comments to what is a already a strong piece of
work even though Dave needed to release it sooner than anticipated.  I
second the recommendation to review [12] for defense strategies.

-paul

Dave Dittrich wrote:
>
> Communication
> -------------
>
> The handler expects commands to be contained entirely in the data
> payload of a single TCP packet, not broken up character by character
> in a stream.  This means that "telnet" cannot be used to control a
> handler, but instead some other client program must be used to buffer
> the command line before sending (e.g., a special command shell or port
> redirector, netcat [19], etc. -- no special client was included in the
> source posted on Security Focus [20]).

Actually, telnet can be used by switching it to line mode.  This mode
is used by telnet clients by default on some OSes.

> Handler Commands
> ----------------
>
> The handler command set is:
>
> help

"commands" is an alias for "help"

> servers
>         List all currently known agents.

The agents may or may not be active.  This is the list of all agents
that have sent the "newserver" command to the handler at some point in
time.

> stream <hostname> <seconds>
>         Begin an attack against a single host, for the specified
>         duration.  The handler resolves the hostname to an IP address
>         and sends the command "mstream/arg1:arg1/arg2" to all agents,
>         where "arg1" is the resolved hosts' IP address twice with a
>         colon between (this simplifies argument parsing in the agent)
>         and "arg2" is the duration in seconds.

The released source uses "mstream/arg1/arg2" without the colon and
repeated arg1.  Is the colon used only in the version in the wild?

> quit
>         Terminates the attacker's connection to the handler.

It also reports the termination to the connected users.


> Agent Commands
> --------------
>
> stream/IP/seconds
>         Starts streaming at IP address for specified duration in
>         seconds.

This does not match the description of the corresponding handler
command above but it does match the released source.

> Password protection
> -------------------
>
> (and can't simply hijack the TCP session, either, because of the command
> buffering described in the Communication section.)

Hmmm.  I don't see why the session can't be hijacked.

> Fingerprints
> ------------
>
> Visible strings in the agent (in two truncated columns to save space)
> are:
>
> ------------------------------------------------------------------------------
> 131.247.208.191                     _end
> 129.79.20.202                       htons@@GLIBC_2.0

Did you perhaps miss these when you were sanitizing?  The second one
is presumably the University of Indiana system mentioned in Appendix
E.

> Bugs in the source code for both handler and agent result in an
> increasing number of raw sockets and UDP sockets in the agent (three
> each are were witnessed on this agent), and an increasing number of
> open file handles and UDP sockets in the handler (hundreds were shown
> by Andrew Korty).

master.c neglects to do a close(fd) at the end of sendtoall and
send2server while server.c neglects to do the same at the end of
send2master.

> The stream2.c attack floods the victim with TCP ACK packets, using
> forged source addresses generated by random() (i.e., any or all
> of the four octets will occasionally be zero) and incrementing source
> port and sequence numbers, as seen in this code snippet:

By stream2.c, I assume that you mean the code sent to BUGTRAQ by Tim
Yardley that he later updated to set the ACK flag (also in [12]).

http://www.securityfocus.com/templates/archive.pike?list=1&date=2000-01-15&msg=4.2.0.58.20000121112253.012a8f10@students.uiuc.edu

The released code server.c contains DoS code ripped almost verbatim
out of stream.c with the ACK flag fix and then modified to hit
multiple targets.  The only other major change is that stream.c gave
the attacker the option to select a specific destination port.
server.c always randomizes the destination port on the victim host.

If you look at Yardley's analysis (also in [12])

http://www.securityfocus.com/templates/archive.pike?list=1&date=2000-01-22&msg=4.2.0.58.20000125093641.0118b5e8@students.uiuc.edu

then you can see that the attack could have potentially worse effect
if the destination port is selected to match an open port on the
victim host.

> Weaknesses
> ----------
>
> Control communication (attacker <-> handler and handler <-> agent)
> is not encrypted, and is thus subject to session hijacking

This is a contradiction of the statement under Password Protection
above.

Other weaknesses:

The handler does not automatically clean up the file containing the
list of known agents.  The stream and mstream commands will blindly
send commands to agents that no longer exist but did at one time.

The code attempts to avoid adding an ip address multiple times to the
list of known agents.  However, a bug (common mistake with continue
and nearest loop) causes the check to fail in its purpose so the same
agent will have its ip address newly recorded in the agent list each
time it notifies the handler with "newserver".  As a result, the
handler will send ping, stream, and mstream commands to an agent
however many times its ip address is recorded in the agent list.

In server.c note the declaration:

  char *ipps[50]

and the code snippet:

  if ((tmpip = strtok(ips, ":")) == NULL) continue;
  ipps[0] = (char *) malloc(strlen(tmpip)+2);
  strncpy(ipps[0], tmpip, strlen(tmpip)+2);
  y = 1;
  while ((tmpip = strtok(NULL, ":")) != NULL) {
    ipps[y] = (char *)malloc(strlen(tmpip)+2);
    strncpy(ipps[y], tmpip, strlen(tmpip)+2);
    y++;
    }
  ipps[y] = NULL;

If an agent receives the mstream command containing a second argument
with greater than 50 colon-separated fields, it will merrily continue
mallocing memory and stuffing the addresses into ipps well past the
array bounds.  Could get interesting when the following code attempts
to free the malloced space:

  for (y = 0 ; ipps[y] != NULL ; y++) free(ipps[y]);

The list could continue but these are the major ones that affect
visible behavior.  Dave was kind when he described the code as more
primitive than other known DDoS tools.

> The next logical evolutionary steps
> -----------------------------------
>
> It is not a stretch to assume that features from other published DDoS
> tools will make their way into tools like mstream.  Briefly, some
> likely enhancements to this code include:

7) Target destination port selection
8) Automated cleanup of known agent list

> Appendix A - References
> -----------------------
>
> [12] BUGTRAQ threads on the stream.c DoS attack and its fallout
>         http://staff.washington.edu/dittrich/misc/ddos/stream.txt
(5057051) ------------------------------------------(Ombruten)

5057579 2000-05-02  23:53  /63 rader/ Postmaster
Mottagare: Bugtraq (import) <10702>
Ärende: Re: Source code to mstream, a DDoS tool
------------------------------------------------------------
Approved-By: aleph1@SECURITYFOCUS.COM
Delivered-To: bugtraq@lists.securityfocus.com
Delivered-To: BUGTRAQ@SECURITYFOCUS.COM
X-Priority: 3
X-MSMail-Priority: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200
Message-ID:  <024001bfb3ea$f612a740$6600a8c0@localarc.com>
Date:         Tue, 2 May 2000 00:00:31 -0400
Reply-To: Security <security@ARC.COM>
Sender: Bugtraq List <BUGTRAQ@SECURITYFOCUS.COM>
From: Security <security@ARC.COM>
X-To:         BUGTRAQ@SECURITYFOCUS.COM
To: BUGTRAQ@SECURITYFOCUS.COM

Based on the signatures provided by Mr. Dittrich, we have updated
SARA (http://www-arc.com/sara) (version 3.0.2) to detect the presence
of the mstream DDOS (both wild and published).

------------------------------------------------------------------
Bob Todd
Advanced Research Corporation
http://www-arc.com

----- Original Message -----
From: Dave Dittrich <dittrich@CAC.WASHINGTON.EDU>
To: <BUGTRAQ@SECURITYFOCUS.COM>
Sent: Monday, May 01, 2000 5:08 PM
Subject: Re: Source code to mstream, a DDoS tool


> ==========================================================================
>
>        The "mstream" distributed denial of service attack tool
>
> ==========================================================================
>
> May 1, 2000
> Copyright (C) 2000. All rights reserved.
>
> David Dittrich
> University of Washington
> <dittrich@cac.washington.edu>
>
> George Weaver
> Pennsylvania State University
> <weaver@gabriel.nso.psu.edu>
>
> Sven Dietrich
> NASA Goddard Space Flight Center
> <spock@netsec.gsfc.nasa.gov>
>
> Neil Long
> Oxford University
> <neil.long@computing-services.oxford.ac.uk>
>
>
> Introduction
> ------------
>
> The following is an analysis of "mstream", a distributed denial of
> service (DDoS) attack tool, based on the source code of "stream2.c", a
> classic point-to-point DoS attack tool [12].
<<<<< cut >>>>>
(5057579) ------------------------------------------