5054580 2000-05-02 00:34 /1864 rader/ Postmaster
Mottagare: Bugtraq (import) <10691>
Ärende: Re: Source code to mstream, a DDoS tool
------------------------------------------------------------
Approved-By: aleph1@SECURITYFOCUS.COM
Delivered-To: bugtraq@lists.securityfocus.com
Delivered-To: bugtraq@securityfocus.com
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Message-ID: <Pine.GUL.4.21.0005011351460.11047-100000@red7.cac.washington.edu>
Date: Mon, 1 May 2000 14:08:15 -0700
Reply-To: Dave Dittrich <dittrich@CAC.WASHINGTON.EDU>
Sender: Bugtraq List <BUGTRAQ@SECURITYFOCUS.COM>
From: Dave Dittrich <dittrich@CAC.WASHINGTON.EDU>
X-To: auscert@auscert.org.au, cert@cert.org, ciac@llnl.gov
submissions@packetstorm.securify.com, bugtraq@securityfocus.com
2600-list@wiretapped.net, contact@hackernews.com
webmaster@2600.com, vuln-dev@securityfocus.com
Incidents Mailing List <INCIDENTS@securityfocus.com>,
oth@2600.com To: BUGTRAQ@SECURITYFOCUS.COM In-Reply-To:
<200004291748.TAA13203@lobeda.jena.thur.de>
==========================================================================
The "mstream" distributed denial of service attack tool
==========================================================================
May 1, 2000
Copyright (C) 2000. All rights reserved.
David Dittrich
University of Washington
<dittrich@cac.washington.edu>
George Weaver
Pennsylvania State University
<weaver@gabriel.nso.psu.edu>
Sven Dietrich
NASA Goddard Space Flight Center
<spock@netsec.gsfc.nasa.gov>
Neil Long
Oxford University
<neil.long@computing-services.oxford.ac.uk>
Introduction
------------
The following is an analysis of "mstream", a distributed denial of
service (DDoS) attack tool, based on the source code of "stream2.c", a
classic point-to-point DoS attack tool [12].
There are currently seven major code bases of recognized DDoS tools
witnessed in use "in the wild" for executing DDoS attacks:
trinoo [03]
Tribe Flood Network (TFN) [04]
Tribe Flood Network 2000 (tfn2k) [06]
stacheldraht/stacheldrahtV4 [05]
stacheldraht v2.666
shaft [07]
mstream
mstream is more primitive than any of the other DDoS tools.
Examination of reverse engineered and recovered C source code reveals
the program to be in early development stages, with numerous bugs and
an incomplete feature set compared with any of the other listed
tools. The effectiveness of the stream/stream2 attack itself,
however, means that it will still be disruptive to the victim (and
agent) networks even with an attack network consisting of only a hand
full of agents.
(The source code for mstream was anonymously posted to the
vuln-dev@securityfocus.com and BUGTRAQ email lists on April 29,
2000. [20] As a result, this analysis is being released in a bit less
polished and complete form that originally intended, but will
hopefully allow incident response teams and vendors to save some time
in developing their responses. Any errors we didn't catch are likely
a result of this rush.)
The reader is advised that modification of the source code can and
would change any of the details of this analysis, such as prompts,
passwords, commands, TCP/UDP port numbers, or supported attack
methods, signatures, and features. In fact, the values of
communication ports have already been seen to differ between
published and "in the wild" code.
[Note also that throughout this analysis, actual nicks,
site names, and IP addresses have been sanitized.]
The handler/agent terminology used in this analysis was developed at
the CERT Distributed System Intruder Tools workshop held in November
1999, and will be used in this analysis. It is highly recommended
that the CERT workshop report [01] be read first, as well as
background and supporting information on other DDoS tools [08]:
http://www.cert.org/reports/dsit_workshop.pdf
http://staff.washington.edu/dittrich/misc/ddos/
An mstream agent was discovered in late April 2000 on a compromised
Linux system at a major university. This system was identified to be
flooding packets using forged source addresses, targeted at over a
dozen IP addresses.
Since RFC 2267 style egress filtering [13] was employed on this
network, and all 32 bits of the source addresses were forged, only an
extremely small number of attack packets (matching the internal
network blocks) could have managed to leave the agent's network. The
traffic did, however, cause the router (which served 18 subnets) to
become non-responsive. This means that sites that do egress
filtering may still suffer from these attacks themselves, even if the
intended "victim" receives fewer packets than the attacker(s)
intended. The lesson here is that there is no "quick fix" to DDoS in
the form of simple technical filtering solutions. [The router
manufacturer was briefed and is currently working on identifying the
cause and fixes.]
Another side-effect of taking egress filtering down one level to that
of internal subnets is that rejected packets will not make it past
the router doing the filtering, so the effects of bandwidth
consumption or router disruption will not be felt above the level of
the router doing the filtering (or being saturated). This means a
border router based IDS (e.g., net flows), or one outside your
borders on a DMZ or upstream ISP's network, will not identify the
attempted attacks. Unless you are monitoring the routers themselves,
only user complaints would tip you off to an attack originating from
your network. It also means that packet level analysis is made more
difficult, as it must be done *in front* of the router doing the
filtering in order to capture all packets. This puts an added burden
on network engineers or incident responders to do packet level
dumping, on site in a router closet, in order to know with a high
degree of reliability what is going on.
A major problem in forensic analysis of these attacks -- whether
successful or blocked by egress filters -- is having policies and
procedures in place to facilitate packet level logging. At least one
site that was a victim of early February 2000 DoS attacks on eCommerce
sites was not prepared for this kind of analysis, and to this day is
not aware of what attack tool was used against them. They only knew
their routers were crashing and relied on their upstream provider to
determine how to block packets to restore throughput. This results in
a major roadblock to investigation by law enforcement, who have been
criticized heavily by some for their slow response (it is a
non-trivial problem to perform an investigation of a DoS or DDoS
attack with nothing but router or browser response behavioral
observations to go on.) A description of the details of the attack on
Yahoo! was published on Packet Storm Security's web site [11], but
not many other details were available. (This was also pointed out by
Anonymous [20] in his/her post.)
Unlike the high volume mass intrusions witnessed with trinoo, TFN,
and stacheldraht in 1999, this particular mstream network appears to
be in the very early stages of code development and to have been set
up by hand (on both handler and agent systems) with a slightly
modified Linux rootkit version 4 [10] to conceal the presence of the
intruder's activity. For a description of intrusion and concealment
on an agent system, see Appendix D. For a description of intrusion
and concealment on a handler system, see Appendix E.
The network: client(s)-->handler(s)-->agent(s)-->victim(s)
------------------------------------------------------------
The mstream network, like trinoo and shaft, is made up of one or more
handlers ("master.c") and a large set of agents ("server.c").
Attacker to handler communication is at present unencrypted over TCP,
with handler <-> agent communication unencrypted over UDP.
An mstream network would look like this:
+----------+ +----------+
| attacker | | attacker |
+----------+ +----------+
| |
. . . --+------+---------------+------+----------------+-- . . .
| | |
| | |
+-----------+ +-----------+ +-----------+
| handler | | handler | | handler |
+-----------+ +-----------+ +-----------+
| | |
| | |
. . . ---+------+-----+------------+---+--------+------------+-+-- . . .
| | | | |
| | | | |
+-------+ +-------+ +-------+ +-------+ +-------+
| agent | | agent | | agent | | agent | | agent |
+-------+ +-------+ +-------+ +-------+ +-------+
Communication
-------------
Attacker to handler(s): 6723/tcp (in published source)
15104/tcp ("in the wild")
12754/tcp (in recovered source)
Agent to Handler(s): 9325/udp (in published source)
6838/udp ("in the wild")
Handler to agent(s): 7983/udp (in published source)
10498/udp ("in the wild")
Remote control of the mstream handler is accomplished via a TCP
connection to port 6723/tcp (or 15104/tcp, or 12754/tcp, or...).
The handler expects commands to be contained entirely in the data
payload of a single TCP packet, not broken up character by character
in a stream. This means that "telnet" cannot be used to control a
handler, but instead some other client program must be used to buffer
the command line before sending (e.g., a special command shell or port
redirector, netcat [19], etc. -- no special client was included in the
source posted on Security Focus [20]).
The traffic over this connection is not encrypted (although it has
been shown in stacheldraht that adding a Blowfish block cipher is not
difficult.) Command lines are space separated argument lists.
Like trinoo, communication between the handler(s) and agent(s) is
accomplished using UDP datagrams. Agent commands are slash ("/")
separated argument lists, with some multi-item arguments being colon
(":") separated lists.
[It is believed that the difference between attacker <-> handler
communication port numbers between "lsof" output on a system running
an active handler and that in the recovered source code is due to
version differences. That is, the recovered source may have been an
older version than the source used to compile the handler actually
running at the time. Both ports are identified here, although they
are trivial to change to something else (and were found to be
different in the published source [20]).]
The examined code limits the maximum number of connected attackers to
3. This may be a protective measure, or possibly a redundant access
measure in case one or more systems used as intermediaries by the
attacker are discovered or otherwise taken out of service.
After connecting, the user must supply the proper password (default
is "N7%diApf!" in the recovered code, and "sex" in the published code
[20]). If the proper password is not given, all currently connected
users are notified of the attempt and the connection is dropped. If
the proper password is given, all currently connected users are
informed of the new session and the user is presented with a "> "
prompt.
Handler Commands
----------------
The handler commands consist of up to 3 space-delimited fields.
If a connected attacker does not enter a command within 420 seconds,
the connection is terminated. (It is assumed "420" was not chosen
simply because 7 minutes is an ideal timeout value. ;)
The handler command set is:
help
Prints the following:
Available commands:
stream stream attack !
servers Prints all known servers.
ping ping all servers.
who tells you the ips of the people logged in
mstream lets you stream more than one ip at a time
servers
List all currently known agents.
who
Shows the currently connected users.
ping
Identify remaining active agents. Sends the command "ping" to
all known agents and reports to the connected users as each
"pong" reply is received.
stream <hostname> <seconds>
Begin an attack against a single host, for the specified
duration. The handler resolves the hostname to an IP address
and sends the command "mstream/arg1:arg1/arg2" to all agents,
where "arg1" is the resolved hosts' IP address twice with a
colon between (this simplifies argument parsing in the agent)
and "arg2" is the duration in seconds.
mstream <ip1:ip2:ip3:...> <seconds>
Begin an attack against multiple IP addresses, for the specified
duration. The handler sends the command "mstream/arg1/arg2" to
all agents, where "arg1" is the list of colon separated IP
addresses, and "arg2" is the duration in seconds. Also for
simplicity, in this command, there is no host name resolution
(i.e., you MUST specify all targets by a properly formed colon
separated list of IP addresses)
quit
Terminates the attacker's connection to the handler.
Agent Commands
--------------
The handler communicates to agents using string based commands
in the data portion of UDP packets. These commands are not encrypted
(although this, too, can easily be changed.)
There are only three agent commands currently. Commands are either
a simple string, or a slash ("/") separated command and argument list.
ping
Replies to IP address that sent this packet with "pong".
stream/IP/seconds
Starts streaming at IP address for specified duration in
seconds.
mstream/IP1[:IP2[:IPN]]/seconds
Starts streaming at all of the colon separated list of IP
addresses for specified duration in seconds.
Even though the agent "in the wild" had two options for accepting
DDoS commands, namely "stream" and "mstream" as in the published
source, only the mstream command is used in the handler to agent
protocol. A simple "stream 192.168.0.100 10" command to the handler
sends the powerful command "mstream/192.168.0.100:192.168.0.100/10"
to the agent, when in fact a simple "stream/192.168.0.100/10" should
be generated. It is not clear why this was done, but it does look
like simple algorithms are used for command parsing, so this might
just indicate a "quick and dirty" development process.
Password protection
-------------------
The handler is password protected, to prevent trivial takeover of
the network handler. The password is not encrypted, just a string
that is compared against the data paylod of the initial packet as-is.
It should be explicitly noted here again that this program has a
feature not found in other DDoS tools, which informs all connected
users of access, sucessful or not, to the handler(s) by competing
parties (black hat or white hat). Thus it does not matter that you
can identify the password string in the binary, since you can't use
it without detection (and can't simply hijack the TCP session,
either, because of the command buffering described in the
Communication section.)
There is no password protection of handler <-> agent communication,
but that isn't surprising. As was seen with trinoo, a password in
clear text is not much of a defense and is trivially attacked by
sniffing network traffic.
Fingerprints
------------
As mentioned above, command strings between the handler(s) and
agent(s) is visible in packet flows.
Visible strings in the agent (in two truncated columns to save space)
are:
------------------------------------------------------------------------------
ELF mstream
/lib/ld-linux.so.2 ping
GNU pong
__gmon_start__ fork
libc.so.6 init.c
random . . .
getpid server.c
perror strchr@@GLIBC_2.0
getuid packet
malloc getpid@@GLIBC_2.0
recvfrom _DYNAMIC
socket _etext
bind __register_frame_info@@GLIBC_2.0
inet_addr recvfrom@@GLIBC_2.0
__deregister_frame_info _fp_hw
setsockopt perror@@GLIBC_2.0
rand fork@@GLIBC_2.0
strncmp sock
strncpy cksum
sendto random@@GLIBC_2.0
strtok _init
fork malloc@@GLIBC_2.0
memset getppid@@GLIBC_2.0
srand sendto@@GLIBC_2.0
getppid __deregister_frame_info@@GLIBC_2.0
time setsockopt@@GLIBC_2.0
htons time@@GLIBC_2.0
exit _start
atoi forkbg
_IO_stdin_used strlen@@GLIBC_2.0
__libc_start_main stream
strlen strncmp@@GLIBC_2.0
strchr inet_addr@@GLIBC_2.0
__register_frame_info __bss_start
free main
GLIBC_2.0 __libc_start_main@@GLIBC_2.0
PTRh data_start
QVh0 bind@@GLIBC_2.0
Ph% getuid@@GLIBC_2.0
PhG _fini
WVS s_in
[^_ srand@@GLIBC_2.0
WVS nlstr
j(j exit@@GLIBC_2.0
j h atoi@@GLIBC_2.0
j(h _edata
j h in_cksum
j(h _GLOBAL_OFFSET_TABLE_
[^_ free@@GLIBC_2.0
131.247.208.191 _end
129.79.20.202 htons@@GLIBC_2.0
socket send2master
bind memset@@GLIBC_2.0
setsockopt strncpy@@GLIBC_2.0
newserver _IO_stdin_used
stream strtok@@GLIBC_2.0
__data_start __gmon_start__
socket@@GLIBC_2.0 rand@@GLIBC_2.0
------------------------------------------------------------------------------
Visible strings in the handler are:
------------------------------------------------------------------------------
% strings -n 3 master Available commands:
socket stream
bind stream attack !
listen servers
setsockopt Prints all known servers.
fcntl ping
You're too idle ! ping all servers.
Connection from %s who
newserver tells you the ips of the people log
New server on %s. mstream
pong lets you stream more than one ip at
Got pong number %d from %s who
%s has disconnected (not auth'd): % Currently Online:
Invalid password from %s. Socket number %d
Password accepted for connection fr [%s]
Lost connection to %s: %s ping
stream Pinging all servers.
Usage: stream <hostname> <seconds> mstream
Unable to resolve %s. Usage: mstream <ip1:ip2:ip3:...> <s
stream/%s/%s MStreaming %s for %s seconds.
Streaming %s for %s seconds. mstream/%s/%s
quit fork
%s has disconnected. Forked into background, pid %d
servers Caught SIGHUP, ignoring.
Server file doesn't exist, creating Caught SIGINT, ignoring.
The following ips are known servers Segmentation Violation, Exiting cle
help Caught unknown signal, This should
commands
------------------------------------------------------------------------------
The agent (named "rpc.wall" on this system -- this same name was used
for the handler as well) is seen with "lsof" as follows:
------------------------------------------------------------------------------
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
rpc.wall 588 root cwd DIR 3,2 1024 2 /
rpc.wall 588 root rtd DIR 3,2 1024 2 /
rpc.wall 588 root txt REG 3,3 17016 15765 /usr/bin/rpc.wall
rpc.wall 588 root mem REG 3,2 342206 30771 /lib/ld-2.1.1.so
rpc.wall 588 root mem REG 3,2 4016683 30789 /lib/libc-2.1.1.so
rpc.wall 588 root 0u CHR 5,1 4952 /dev/console
rpc.wall 588 root 1w FIFO 0,0 646 pipe
rpc.wall 588 root 2w FIFO 0,0 647 pipe
rpc.wall 588 root 3u IPv4 656 UDP *:10498
rpc.wall 588 root 4u IPv4 657 UDP *:1044
rpc.wall 588 root 5u IPv4 658 UDP *:1045
rpc.wall 588 root 6u raw 30219 00000000:00FF->00000000:0000 st=07
rpc.wall 588 root 7r FIFO 0,0 648 pipe
rpc.wall 588 root 8u raw 30241 00000000:00FF->00000000:0000 st=07
rpc.wall 588 root 9u CHR 5,1 4952 /dev/console
rpc.wall 588 root 10u IPv4 30244 UDP *:1051
rpc.wall 588 root 11u raw 30245 00000000:00FF->00000000:0000 st=07
rpc.wall 588 root 21w FIFO 0,0 648 pipe
------------------------------------------------------------------------------
Bugs in the source code for both handler and agent result in an
increasing number of raw sockets and UDP sockets in the agent (three
each are were witnessed on this agent), and an increasing number of
open file handles and UDP sockets in the handler (hundreds were shown
by Andrew Korty). [This is no doubt an indication that mstream is in
the early development stages, so this signature should not be counted
on to identify a handler or agent running on a system.]
When an agent first starts up, it sends a "newserver" command to
the list of default handlers compiled into it, as seen here with
tcpdump:
----------------------------------------------------------------------------
00:04:38.530000 192.168.0.20.1081 > 192.168.0.100.6838: udp 9
0x0000 4500 0025 ef75 0000 4011 098a c0a8 0014 E..%.u..@.......
0x0010 c0a8 0064 0439 1ab6 0011 2b63 6e65 7773 ...d.9....+cnews
0x0020 6572 7665 7200 0000 0000 0000 0000 erver.........
----------------------------------------------------------------------------
If a rootkit is in place (as it was on both handler and agent
systems), you cannot trust the standard operating system
commands to show you the running handler or agent, or their
network connections. The main lesson to be learned from rootkits is
that a large percentage of Unix system administrators will NOT be
skilled enough to get around rootkits. This means a couple things.
First, and fundamentally, intruders will tend to have an even greater
advantage over unskilled system administrators. It is becoming ever
more important that systems administrators -- Unix, NT, whatever --
have training as a primary task, not a luxury or burden to be
avoided. The moment of a security incident is NOT the proper time to
catch up on months of missed learning, and causes undue pressure to
take shortcuts to get the system back up quickly (usually making
things MUCH worse).
Second, incident response and forensic investigation may be made more
difficult, if not impossible, as the simple "solution" that the
unskilled Unix administrator will take is to give up an just
re-install the operating system. This ill-advised choice of action
destroys any evidence that may exist on the system and sets the
system up for a subsequent intrusion because the same security
precautions they did not take before the incident will usually not be
taken this time either. All too often, this action is taken without
first seeking advice and before incident response teams are able to
notify the administrator and assist them in taking the correct steps.
All system administrators are urged to take the time now to prepare
to deal with rootkits [10].
As mentioned above, if an agent receives a UDP packet on port
10498/udp, containing the string "ping" as its data payload (and if it
is not actively streaming at the time), it will reply to the sending
system with a UDP packet on port 6838/udp with the string "pong" as
its data payload. (The trailing zeroes are just a tcpdump artifact.
The payload is clearly 4 bytes.)
----------------------------------------------------------------------------
00:05:16.457239 192.168.0.100.65364 > 192.168.0.20.10498: udp 5
0x0000 4500 0021 f412 0000 4011 04f1 c0a8 0064 E..!....@......d
0x0010 c0a8 0014 ff54 2902 000d 6ce3 7069 6e67 .....T)...l.ping
0x0020 0a .
00:05:16.458214 192.168.0.20.1083 > 192.168.0.100.6838: udp 4
0x0000 4500 0020 ef8c 0000 4011 0978 c0a8 0014 E.......@..x....
0x0010 c0a8 0064 043b 1ab6 000c 8045 706f 6e67 ...d.;.....Epong
0x0020 0000 0000 0000 0000 0000 0000 0000 ..............
----------------------------------------------------------------------------
This sequence allows signature matching with programs like "ngrep"
[14], "snort" [18] (see Appendix B for snort rules), or scanning for
idle agents using "rid" [15] (see Appendix C for a RID template).
The ngrep command string to detect these packets would be:
# ngrep "p[oi]ng" udp port 6838 or udp port 10498
(You will need to tailor this command, or add the other ports listed
above from the published code, to decrease the chance of false
negatives.)
Attack packets have a fixed size of 40 bytes per packet, which may be
on purpose to evade large packet triggers that exist on some IDSs.
The stream2.c attack floods the victim with TCP ACK packets, using
forged source addresses generated by random() (i.e., any or all
of the four octets will occasionally be zero) and incrementing source
port and sequence numbers, as seen in this code snippet:
------------------------------------------------------------------------------
. . .
for(i=0;;++i) {
cksum.pseudo.saddr = packet.ip.ip_src.s_addr = random();
++packet.ip.ip_id;
++packet.tcp.th_sport;
++packet.tcp.th_seq;
if (!dstport)
s_in.sin_port = packet.tcp.th_dport = rand();
. . .
------------------------------------------------------------------------------
During an attack, the following packet signature would be observed
(as seen by tcpdump using a recovered agent binary):
----------------------------------------------------------------------------
01:39:24.701083 192.168.0.2.65527 > 192.168.0.20.10498: [bad udp cksum 3100!]
udp 24 (ttl 64, id 886)
0x0000 4500 0034 0376 0000 4011 f5dc c0a8 0002 E..4.v..@.......
0x0010 c0a8 0014 fff7 2902 0020 556c 7374 7265 ......)...Ulstre
0x0020 616d 2f31 3932 2e31 3638 2e30 2e31 3030 am/192.168.0.100
0x0030 2f31 300a /10.
01:40:10.132724 192.168.0.2.65526 > 192.168.0.20.10498: [bad udp cksum 3100!]
udp 24 (ttl 64, id 930)
0x0000 4500 0034 03a2 0000 4011 f5b0 c0a8 0002 E..4....@.......
0x0010 c0a8 0014 fff6 2902 0020 556d 7374 7265 ......)...Umstre
0x0020 616d 2f31 3932 2e31 3638 2e30 2e31 3030 am/192.168.0.100
0x0030 2f31 300a /10.
01:41:23.674796 192.168.0.2.65525 > 192.168.0.20.10498: [bad udp cksum 4a00!]
udp 49 (ttl 64, id 1031)
0x0000 4500 004d 0407 0000 4011 f532 c0a8 0002 E..M....@..2....
0x0010 c0a8 0014 fff5 2902 0039 a9b4 6d73 7472 ......)..9..mstr
0x0020 6561 6d2f 3139 322e 3136 382e 302e 313a eam/192.168.0.1:
0x0030 3139 322e 3136 382e 302e 3130 303a 3139 192.168.0.100:19
0x0040 322e 3136 382e 302e 322f 3130 0a 2.168.0.2/10.
01:41:23.675771 arp who-has 192.168.0.1 tell 192.168.0.20
0x0000 0001 0800 0604 0001 0010 5a99 6544 c0a8 ..........Z.eD..
0x0010 0014 0000 0000 0000 c0a8 0001 0000 0000 ................
0x0020 0000 0000 0000 0000 0000 0000 0000 ..............
01:41:23.675772 arp who-has 192.168.0.100 tell 192.168.0.20
0x0000 0001 0800 0604 0001 0010 5a99 6544 c0a8 ..........Z.eD..
0x0010 0014 0000 0000 0000 c0a8 0064 0000 0000 ...........d....
0x0020 0000 0000 0000 0000 0000 0000 0000 ..............
01:41:23.675773 77.172.43.85.38444 > 192.168.0.2.26296: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 50237)
0x0000 4508 0028 c43d 0000 ff06 bdde 4dac 2b55 E..(.=......M.+U
0x0010 c0a8 0002 962c 66b8 ea97 d237 0000 0000 .....,f....7....
0x0020 5010 4000 7c74 0000 0000 0000 0000 P.@.|t........
01:41:23.675774 88.148.222.45.39212 > 192.168.0.2.10342: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 51005)
0x0000 4508 0028 c73d 0000 ff06 fd1d 5894 de2d E..(.=......X..-
0x0010 c0a8 0002 992c 2866 ed97 d237 0000 0000 .....,(f...7....
0x0020 5010 4000 f705 0000 0000 0000 0000 P.@...........
01:41:23.675775 0.18.219.113.39980 > 192.168.0.2.41622: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 51773)
0x0000 4508 0028 ca3d 0000 ff06 555c 0012 db71 E..(.=....U\...q
0x0010 c0a8 0002 9c2c a296 f097 d237 0000 0000 .....,.....7....
0x0020 5010 4000 d213 0000 0000 0000 0000 P.@...........
01:41:23.675776 121.161.140.109.40748 > 192.168.0.2.16749: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 52541)
0x0000 4508 0028 cd3d 0000 ff06 27d1 79a1 8c6d E..(.=....'.y..m
0x0010 c0a8 0002 9f2c 416d f397 d237 0000 0000 .....,Am...7....
0x0020 5010 4000 02b2 0000 0000 0000 0000 P.@...........
01:41:23.675777 79.238.213.72.41516 > 192.168.0.2.46276: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 53309)
0x0000 4508 0028 d03d 0000 ff06 05a9 4fee d548 E..(.=......O..H
0x0010 c0a8 0002 a22c b4c4 f697 d237 0000 0000 .....,.....7....
0x0020 5010 4000 6a32 0000 0000 0000 0000 P.@.j2........
01:41:23.675778 104.24.203.64.42284 > 192.168.0.2.61623: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 54077)
0x0000 4508 0028 d33d 0000 ff06 f486 6818 cb40 E..(.=......h..@
0x0010 c0a8 0002 a52c f0b7 f997 d237 0000 0000 .....,.....7....
0x0020 5010 4000 1a1d 0000 0000 0000 0000 P.@...........
01:41:23.675779 37.60.73.50.43052 > 192.168.0.2.51311: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 54845)
0x0000 4508 0028 d63d 0000 ff06 b671 253c 4932 E..(.=.....q%<I2
0x0010 c0a8 0002 a82c c86f fc97 d237 0000 0000 .....,.o...7....
0x0020 5010 4000 0150 0000 0000 0000 0000 P.@..P........
01:41:23.675780 142.14.73.40.43820 > 192.168.0.2.8979: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 55613)
0x0000 4508 0028 d93d 0000 ff06 4aa9 8e0e 4928 E..(.=....J...I(
0x0010 c0a8 0002 ab2c 2313 ff97 d237 0000 0000 .....,#....7....
0x0020 5010 4000 37e4 0000 0000 0000 0000 P.@.7.........
01:41:23.676748 144.19.212.69.44588 > 192.168.0.2.51668: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 56381)
0x0000 4508 0028 dc3d 0000 ff06 ba86 9013 d445 E..(.=.........E
0x0010 c0a8 0002 ae2c c9d4 0298 d237 0000 0000 .....,.....7....
0x0020 5010 4000 fdff 0000 0000 0000 0000 P.@...........
01:41:23.676749 155.176.45.2.45356 > 192.168.0.2.32793: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 57149)
0x0000 4508 0028 df3d 0000 ff06 532d 9bb0 2d02 E..(.=....S-..-.
0x0010 c0a8 0002 b12c 8019 0598 d237 0000 0000 .....,.....7....
0x0020 5010 4000 dd61 0000 0000 0000 0000 P.@..a........
01:41:23.676750 10.98.211.13.46124 > 192.168.0.2.1995: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 57917)
0x0000 4508 0028 e23d 0000 ff06 3b70 0a62 d30d E..(.=....;p.b..
0x0010 c0a8 0002 b42c 07cb 0898 d237 0000 0000 .....,.....7....
0x0020 5010 4000 3af3 0000 0000 0000 0000 P.@.:.........
01:41:23.676751 214.235.187.89.46892 > 192.168.0.2.14172: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 58685)
0x0000 4508 0028 e53d 0000 ff06 839a d6eb bb59 E..(.=.........Y
0x0010 c0a8 0002 b72c 375c 0b98 d237 0000 0000 .....,7\...7....
0x0020 5010 4000 508c 0000 0000 0000 0000 P.@.P.........
01:41:23.676752 90.193.127.8.47660 > 192.168.0.2.64812: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 59453)
0x0000 4508 0028 e83d 0000 ff06 3916 5ac1 7f08 E..(.=....9.Z...
0x0010 c0a8 0002 ba2c fd2c 0e98 d237 0000 0000 .....,.,...7....
0x0020 5010 4000 3d37 0000 0000 0000 0000 P.@.=7........
01:41:23.676753 160.176.42.60.48428 > 192.168.0.2.17432: . [tcp sum ok]
ack 0 win 16384 [tos 0x8] (ttl 255, id 60221)
0x0000 4508 0028 eb3d 0000 ff06 44f3 a0b0 2a3c E..(.=....D...*<
0x0010 c0a8 0002 bd2c 4418 1198 d237 0000 0000 .....,D....7....
0x0020 5010 4000 ff28 0000 0000 0000 0000 P.@..(........
----------------------------------------------------------------------------
An attack (only packets including "0" octets are shown) would
similarly be seen with Cisco Net Flows like this:
------------------------------------------------------------------------------
% grep "[ \.]0[ \.(]" ddos-000415
Apr 15 04:12:08 tcp 82.0.151.5(29497) -> 192.168.10.5(27072), 1 packet
Apr 15 04:12:18 tcp 207.0.149.32(21893) -> 192.168.10.5(3913), 1 packet
Apr 15 04:12:33 tcp 0.147.151.82(10473) -> 10.4.152.237(2810), 1 packet
Apr 15 04:13:39 tcp 60.0.33.36(41079) -> 10.4.152.237(31754), 1 packet
Apr 15 04:14:03 tcp 103.140.148.0(4247) -> 10.4.152.237(29689), 1 packet
Apr 15 04:14:15 tcp 214.1.99.0(46714) -> 10.4.152.237(22524), 1 packet
Apr 15 04:15:11 tcp 10.148.60.0(12276) -> 192.168.10.5(31122), 1 packet
Apr 15 04:15:20 tcp 0.112.67.108(4550) -> 192.168.10.5(63787), 1 packet
Apr 15 04:15:33 tcp 13.0.16.2(39092) -> 10.4.152.237(57998), 1 packet
. . . Apr 15 06:45:24 tcp 18.167.171.0(54104) -> 10.200.5.8(32779),
1 packet Apr 15 06:45:52 tcp 0.23.15.38(45621) -> 10.200.5.8(20780),
1 packet Apr 15 06:46:14 tcp 0.12.109.77(38670) -> 10.200.5.8(47776),
1 packet Apr 15 07:19:12 tcp 199.120.0.72(64912) ->
10.4.152.237(45151), 1 packet Apr 15 07:27:37 tcp 0.28.232.21(52533)
-> 10.4.152.237(338), 1 packet Apr 15 07:28:13 tcp 99.61.233.0(20951)
-> 10.4.152.237(58427), 1 packet Apr 15 07:31:23 tcp
195.0.3.111(17193) -> 10.4.152.237(14601), 1 packet Apr 15 07:32:19
tcp 61.108.245.0(24309) -> 10.4.152.237(32809), 1 packet
------------------------------------------------------------------------------
It should also be noted that some of the forged source addresses are
broadcast addresses, multicast addresses, or network addresses,
which can have ramifications if packets are directed back to the
flooding systems (see [12]).
Analysis of the de-compiled agent source code shows the stream2.c
attack is altered slightly to randomize more header fields, with some
static values that can be noted when analyzing network traffic:
------------------------------------------------------------------------------
packet.ip.ip_id = rand();
. . .
packet.tcp.th_win = htons(16384);
. . .
packet.tcp.th_seq = random();
. . .
packet.tcp.th_sport = rand();
packet.tcp.th_dport = rand();
. . .
while (time(0) <= endtime) {
if (floodtype != 0) {
i = 0;
while (arg4[i] != NULL) { /* until list exhausted */
if (strchr(arg4[i],'.') != NULL) { /* valid ip */
packet.ip.ip_dst.s_addr = inet_addr(arg4[i]);
cksum.pseudo.daddr = inet_addr(arg4[i]);
s_sin.sin_addr.s_addr = inet_addr(arg4[i]);
cksum.pseudo.saddr = packet.ip.ip_src.s_addr = random();
packet.ip.ip_id++;
packet.tcp.th_sport++;
packet.tcp.th_seq++;
s_in.sin_port = packet.tcp.th_dport = rand();
. . .
}
}
}
}
------------------------------------------------------------------------------
Defenses
--------
There are none. We are all doomed. Time to shop for property in
Montana and stock up on non-perishable food items and semi-automatic
weapons.
(Seriously, we didn't have time to finish this section. Any defenses
already discussed for stream/stream2 [12] or other DDoS tools [08]
would apply. And for God's sake, SECURE ALL THOSE SYSTEMS they are
using to build these networks! ;)
Weaknesses
----------
Control communication (attacker <-> handler and handler <-> agent)
is not encrypted, and is thus subject to session hijacking and
third-party control, respectively. Agents do not authenticate the
source of commands, so you can easily use the "ping"/"pong" feature to
detect agents that are not currently streaming.
In fact, "pong" is the only response sent back to the handler, and
"newserver" is only sent to handler(s) when the program first starts.
(The installation of the agent described in Appendix D causes it to
be run after each boot. This would mean a set of "newserver" packets
could be detected at this point, although it is not recommended that
rebooting be used as the first step in verifying an agent
installation. It is better to follow the more thorough forensic
analysis techniques detailed in Appendices D and E to ensure as
little evidence as possible is deleted or altered.)
The agent will segmentation fault upon receiving a badly formatted
command, e.g. "stream foo bar". It is conceivable that such
fault generation can disable agents, but they will most likely get
restarted at the next reboot of the host.
Flooding an agent with requests will saturate its file handle limit
and cause it to become unresponsive. Since no notification or
acknowledgement is made to the handler, one can easily send >1024
commands to the agent. Most floods are short, so it may be possible to
"squeeze in there".
The agent process is single-threaded, implying that it will not
process incoming commands while in mid-flood. This renders any ideas
about remotely crashing or halting a flooding agent unworkable.
The agent also will, in multi-flood mode, flood all hosts in the
colon-delimited list equally for the same amount of time. This seems
to either implicate packet loss due to congestion or require that
multiple flooding machines with different flood durations were in play
to explain evidence witnessed in argus flows at third party sites,
showing the side-effects of multiple targets being hit at the same
time with spoofed IPs (namely ICMP "Host Unreachable" packets and TCP
RST packets).
One thing that has been witnessed is that when there are multiple
targets which start at the same time there is a tailing-off with some
targets being hit longer than others and the rate of spoofed packets
also seems to decline with time - sort of dwindling. (We ran out
of time to analyze third-party effects of flooding, but there will
no doubt be followup on this topic on BUGTRAQ.)
The next logical evolutionary steps
-----------------------------------
It is not a stretch to assume that features from other published DDoS
tools will make their way into tools like mstream. Briefly, some
likely enhancements to this code include:
1) Adding source port filtering for connections to handler.
2) Adding authentication between handler and agent.
3) Packet size selection.
4) Flags for flood packets (ACK, RST, NUL, random, whatever).
5) Encryption for attacker<->handler traffic.
6) More obfuscation of embedded commands.
FIN
Credits
-------
As principal author of this particular analysis, I guess I get to
do the shouts, greetz, and mad props.
Since last fall, I've spent every major holiday analyzing a DDoS tool.
Halloween (trinoo), Thanksgiving (TFN), Christmas (stacheldraht) and
New Years (building/testing scanning tools for all three). Sven
Dietrich (no, we are not related and are not the same person as some
have mistaken! ;) took the lead on analyzing "shaft" [07] along with
Neil Long, allowing me to actually relax a bit on President's and
Valentine's days.
Then some DDoS attacks brought mstream out of the shadows, and over
the Easter Holiday weekend George Weaver kicked butt hand-decompiling
the agent, Sven Dietrich obtaining packet and system call traces on a
test network, and Neil Long gathering third-party effects data to
correlate with actual attacks, while I analyzed forensic data
gathered from an agent system and tried to keep up with the rest.
Its amazing what can get done when four people consume that much
sweets and hard boiled eggs.
Credit also goes to Andrew Korty and investigators at Indiana
University for their forensic analysis and data gathering. Site to
site cooperation and coordination is key to incident response in DDoS
attacks, and the more people who know about this and learn the
techniques, the better off we'll all be.
I should also thank Anonymous, who posted the source code for mtsream
through Security Focus [20], acknowledging me by name for my earlier
work. My only reservation is that the result of publishing the
source (instead of sending it to me directly) was to alter all of our
weekend plans to rush this analysis out (e.g., I was planning on
doing a training hike all day Sunday.) Whoever you are, I should
think you owe me at least a sixer of good microbrew for that! ;)
(The upside is that this simplifies the problem of getting code into
the hands of all vendors, incident response teams, and security
software authors in a fair and above board way.)
--
Dave Dittrich <dittrich@cac.washington.edu>
University of Washington
Appendix A - References
-----------------------
[00] TCP/IP Illustrated, Vol. I, II, and III. W. Richard Stevens and Gary
R. Wright., Addison-Wesley
[01] CERT Distributed System Intruder Tools Workshop report
http://www.cert.org/reports/dsit_workshop.pdf
[02] CERT Advisory CA-99-17 Denial-of-Service Tools
http://www.cert.org/advisories/CA-99-17-denial-of-service-tools.html
[03] The DoS Project's "trinoo" distributed denial of service attack
tool, David Dittrich
http://staff.washington.edu/dittrich/misc/trinoo.analysis
[04] The "Tribe Flood Network" distributed denial of service attack
tool, David Dittrich
http://staff.washington.edu/dittrich/misc/tfn.analysis
[05] The "stacheldraht" distributed denial of service attack tool,
David Dittrich
http://staff.washington.edu/dittrich/misc/stacheldraht.analysis
[06] TFN2K - An Analysis, Jason Barlow and Woody Thrower, Axent
Security Team
http://packetstorm.securify.com/distributed/TFN2k_Analysis-1.3.txt
[07] An analysis of the ``Shaft'' distributed denial of service tool,
Sven Dietrich, Neil Long, and David Dittrich
http://netsec.gsfc.nasa.gov/~spock/shaft_analysis.txt
[08] Distributed Denial of Service (DDoS) Attack Tools, David Dittrich
http://staff.washington.edu/dittrich/misc/ddos/
[09] Distributed denial of service attack tools at Packet Storm Security
http://packetstorm.securify.com/distributed/
[10] "Root Kits" and hiding files/directories/processes after a break-in,
David Dittrich
http://staff.washington.edu/dittrich/misc/faqs/rootkits.faq
[11] Technical details of the attack on Yahoo!
http://packetstorm.securify.com/distributed/yahoo.txt
[12] BUGTRAQ threads on the stream.c DoS attack and its fallout
http://staff.washington.edu/dittrich/misc/ddos/stream.txt
[13] RFC 2267 -- Network Ingress Filtering: Defeating Denial of Service
Attacks which employ IP Source Address Spoofing, Paul Fergussen
and Daniel Senie
ftp://ftp.isi.edu/in-notes/rfc2267.txt
[14] ngrep
http://www.packetfactory.net/ngrep/
[15] rid
http://theorygroup.com/Software/RID
[16] Dan Farmer & Wietse Venema's class on computer forensic analysis
http://www.fish.com/security/forensics.html
[17] tcpdump
ftp://ftp.ee.lbl.gov/tcpdump.tar.Z
[18] snort
http://www.clark.net/~roesch/security.html
[19] netcat ("nc"), Hobbit
http://packetstorm.securify.com/UNIX/netcat/nc110.tgz
[20] Source code for mstream
http://securityfocus.com/templates/archive.pike?list=82&date=2000-04-29&thread=200004291748.TAA13203@lobeda.jena.thur.de
Appendix B - Example snort rules for detecting mstream
------------------------------------------------------
alert UDP any any -> any 6838 (msg:
"IDS100/ddos-mstream-agent-to-handler"; content: "newserver"; ) alert
UDP any any -> any 10498 (msg:
"IDS101/ddos-mstream-handler-to-agent"; content: "stream/"; ) alert
UDP any any -> any 10498 (msg:
"IDS102/ddos-mstream-handler-ping-to-agent" ; content: "ping";) alert
UDP any any -> any 10498 (msg:
"IDS103/ddos-mstream-agent-pong-to-handler" ; content: "pong";) alert
TCP any any -> any 12754 (msg:
"IDS109/ddos-mstream-client-to-handler"; flags: S;) alert TCP any
12754 -> any any (msg: "IDS110/ddos-mstream-handler-to-client";
content: ">"; flags: AP;) alert TCP any any -> any 15104 (msg:
"IDS111/ddos-mstream-client-to-handler"; flags: S;) alert TCP any
15104 -> any any (msg: "IDS112/ddos-mstream-handler-to-client";
content: ">"; flags: AP;)
Appendix C - Rid templates for detecting mstream
------------------------------------------------
start mstream-wild
send udp dport=10498 data="ping"
recv udp dport=6838 data="pong" nmatch=2
end mstream-wild
start mstream-published
send udp dport=7983 data="ping"
recv udp dport=9325 data="pong" nmatch=2
end mstream-published
Appendix D - Initial Intrusion and Concealment on Agent system
--------------------------------------------------------------
Examination of an agent system, and interviewing the owner, identified
what looks like possibly two separate compromises. One sometime
before March 31, 2000 (a password file entry for "inertia" existed
until removed by the system owner on April 1), the transfer and
installation of a rootkit (lrk4) and DDoS agent ("rpc.wall") on April
13 16:02 (all times, unless otherwise noted, are US/Pacific, or
GMT-0700), and the marks of an ADM named attack and login on April 15
05:55. System logs had been deleted and/or scrubbed, so log evidence
was not useful in determining what occurred on the system.
Some of these activities can still be identified using the "mactime"
program, part of Dan Farmer and Wietse Venema's "Coroner's Toolkit"
[16]. [The date "100" is, obviously, a Perl Y2K bug in the "mactime"
program. This software was graciously provided by Dan Farmer, even
though it is not publicly available at this time. It definitely
shows how useful Unix forensic analysis tools can be.]
------------------------------------------------------------------------------
Apr 13 100 16:02:42 12060 .aa -rwxr-xr-x root/www root /bin/chown
12660 m.m -r-sr-xr-x root/www bin /bin/login
Apr 13 100 16:02:43 2048 mcmc drwxr-xr-x root/www root /bin
12660 cc -r-sr-xr-x root/www bin /bin/login
168748 .a. -rwxr-xr-x root/www root /usr/bin/as
64796 .a. -rwxr-xr-x root/www root /usr/bin/egcs
64796 .a. -rwxr-xr-x root/www root /usr/bin/gcc
64796 .a. -rwxr-xr-x root/www root /usr/bin/i386-redhat-linux-gcc
168496 .a. -rwxr-xr-x root/www root /usr/bin/ld
12656 m.c -rws--x--x root/www root /usr/bin/old
12656 m.c -r-xr-xr-x root/www bin /usr/bin/xstat
2315 .a. -rw-r--r-- root/www root /usr/include/_G_config.h
1313 .a. -rw-r--r-- root/www root /usr/include/alloca.h
4090 .a. -rw-r--r-- root/www root /usr/include/arpa/inet.h
3451 .a. -rw-r--r-- root/www root /usr/include/bits/byteswap.h
13327 .a. -rw-r--r-- root/www root /usr/include/bits/confname.h
168 .a. -rw-r--r-- root/www root /usr/include/bits/endian.h
2283 .a. -rw-r--r-- root/www root /usr/include/bits/errno.h
5107 .a. -rw-r--r-- root/www root /usr/include/bits/fcntl.h
4647 .a. -rw-r--r-- root/www root /usr/include/bits/in.h
3406 .a. -rw-r--r-- root/www root /usr/include/bits/posix_opt.h
2842 .a. -rw-r--r-- root/www root /usr/include/bits/select.h
4673 .a. -rw-r--r-- root/www root /usr/include/bits/sigset.h
1716 .a. -rw-r--r-- root/www root /usr/include/bits/sockaddr.h
9033 .a. -rw-r--r-- root/www root /usr/include/bits/socket.h
1297 .a. -rw-r--r-- root/www root /usr/include/bits/stdio_lim.h
2015 .a. -rw-r--r-- root/www root /usr/include/bits/time.h
4673 .a. -rw-r--r-- root/www root /usr/include/bits/types.h
1781 .a. -rw-r--r-- root/www root /usr/include/bits/uio.h
1798 .a. -rw-r--r-- root/www root /usr/include/endian.h
2481 .a. -rw-r--r-- root/www root /usr/include/errno.h
4579 .a. -rw-r--r-- root/www root /usr/include/fcntl.h
9433 .a. -rw-r--r-- root/www root /usr/include/features.h
5861 .a. -rw-r--r-- root/www root /usr/include/getopt.h
973 .a. -rw-r--r-- root/www root /usr/include/gnu/stubs.h
10291 .a. -rw-r--r-- root/www root /usr/include/libio.h
17327 .a. -rw-r--r-- root/www root /usr/include/netdb.h
10779 .a. -rw-r--r-- root/www root /usr/include/netinet/in.h
1591 .a. -rw-r--r-- root/www root /usr/include/netinet/in_systm.h
9086 .a. -rw-r--r-- root/www root /usr/include/netinet/ip.h
4855 .a. -rw-r--r-- root/www root /usr/include/netinet/tcp.h
2550 .a. -rw-r--r-- root/www root /usr/include/rpc/netdb.h
6467 .a. -rw-r--r-- root/www root /usr/include/stdint.h
20816 .a. -rw-r--r-- root/www root /usr/include/stdio.h
27654 .a. -rw-r--r-- root/www root /usr/include/stdlib.h
13245 .a. -rw-r--r-- root/www root /usr/include/string.h
2104 .a. -rw-r--r-- root/www root /usr/include/strings.h
4932 .a. -rw-r--r-- root/www root /usr/include/sys/cdefs.h
3359 .a. -rw-r--r-- root/www root /usr/include/sys/select.h
7996 .a. -rw-r--r-- root/www root /usr/include/sys/socket.h
1577 .a. -rw-r--r-- root/www root /usr/include/sys/sysmacros.h
5337 .a. -rw-r--r-- root/www root /usr/include/sys/time.h
5299 .a. -rw-r--r-- root/www root /usr/include/sys/types.h
1907 .a. -rw-r--r-- root/www root /usr/include/sys/uio.h
9314 .a. -rw-r--r-- root/www root /usr/include/time.h
36708 .a. -rw-r--r-- root/www root /usr/include/unistd.h
874 .a. -rw-r--r-- root/www root /usr/lib/crtn.o
1446620 .a. -rwxr-xr-x root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/cc1
46816 .a. -rwxr-xr-x root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/collect2
88444 .a. -rwxr-xr-x root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/cpp
1424 .a. -rw-r--r-- root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/crtend.o
5794 .a. -rw-r--r-- root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/include/stdarg.h
9834 .a. -rw-r--r-- root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/include/stddef.h
770000 .a. -rw-r--r-- root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/libgcc.a
1957 .a. -rw-r--r-- root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/specs
178 .a. -rw-r--r-- root/www root /usr/lib/libc.so
69638 .a. -rw-r--r-- root/www root /usr/lib/libc_nonshared.a
6162 .a. -rw-r--r-- 1046 squid /usr/src/linux/include/asm-i386/errno.h
1492 .a. -rw-r--r-- 1046 squid /usr/src/linux/include/asm-i386/socket.h
277 .a. -rw-r--r-- 1046 squid /usr/src/linux/include/asm-i386/sockios.h
305 .a. -rw-r--r-- 1046 squid /usr/src/linux/include/linux/errno.h
Apr 13 100 16:02:44 702 mcmc -rwxr-xr-x root/www root /etc/rc.d/rc.local
1024 mcmc drwxr-xr-x root/www root /root/.ncftp
9 mcmc lrwxrwxrwx root/www root /root/.ncftp/history
9 mcmc lrwxrwxrwx root/www root /root/.ncftp/log
9 mcmc lrwxrwxrwx root/www root /root/.ncftp/trace
29696 m.c drwxr-xr-x root/www root /usr/bin
17016 m.c -rwxr-xr-x root/www root /usr/bin/rpc.wall
8460 .a. -rw-r--r-- root/www root /usr/lib/crt1.o
1124 .a. -rw-r--r-- root/www root /usr/lib/crti.o
1892 .a. -rw-r--r-- root/www root /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/crtbegin.o
. . .
Apr 15 100 05:55:09 1024 mcmc drwxr-xr-x root/www root /var/named
1024 mcmc drwxr-xr-x root/www root /var/named/ADMROCKS
Apr 15 100 05:56:19 20437 .a. -rwxr-xr-x root/www root /usr/sbin/tcpd
Apr 15 100 05:56:20 34 .aa -rw-r--r-- root/www root /usr/libexec/awk/addy.awk
35628 .a. -rwxr-xr-x root/www root /usr/sbin/in.telnetd
Apr 15 100 05:56:26 159576 .a. -rwxr-xr-x root/www root /usr/bin/pico
975 .a. -rw-r--r-- root/www root /usr/share/terminfo/v/vt200
975 .a. -rw-r--r-- root/www root /usr/share/terminfo/v/vt220
------------------------------------------------------------------------------
From this listing, the following observations can be made:
o On April 13 at 16:02, /bin/login was created and /bin/chown run.
o At the same time, the compile was run (gcc/egcs) and /bin/old and
/bin/xstat created. Access times on the .h files listed show the
program being compiled uses network libraries.
o Next, the /etc/rc.d/rc.local file is modified to include the
line "/usr/bin/rpc.wall" at the end, thus re-starting the agent on
each reboot. ncftp logging files are modified (they were deleted
and turned into links to /dev/null to disable logging of ncftp
file transfers.)
o The program /usr/bin/rpc.wall was modified, and C runtime libraries
accessed, which implies it was run. (This cannot be confirmed
because the rootkit prevented the program from being seen by
the administrator, who also rebooted the system and restarted it,
thus modifying the /usr/bin/rpc.wall access date.)
o On April 15 at 05:55, it appears that the ADM named buffer overrun
exploit was used against this system, followed within the minute
access to a tcpd wrapped service which invoked in.telnetd.
The rootkit configuration file /usr/libexec/awk/addy.awk was
also accessed. [It is not clear if this was a just a coincidental
second (third?) intrusion attempt.]
o Six seconds later /usr/bin/pico was run and the vt200 terminfo
definition was accessed. Since the trojan version of login contains
the string "vt200", this confirms the backdoor that was installed
two days earlier was used to gain root access.
While the /etc/passwd entry for the "inertia" account had been deleted
by the administrator of the system, the /etc/shadow entry (original
modification date not known) was not:
------------------------------------------------------------------------------
inertia:iUCNir1cd8pI2:::::::
------------------------------------------------------------------------------
The password, "hi", was cracked in a fraction of a second.
Strings in "/bin/login" show the classic trojan horse signs of a path
to a non-standard program ("/usr/bin/xstat") and embedded terminal
type ("vt200") that triggers a root shell:
------------------------------------------------------------------------------
. . .
login
/bin/sh
/usr/bin/xstat
TERM
bcshjvmudzwxftejk
vt200
%s=%s
init.c
. . .
------------------------------------------------------------------------------
Strings in "/bin/ls" and "/bin/ps" show the names of the rootkit
configuration files to be "/usr/libexec/awk/files.awk" and
"/usr/libexec/awk/ps.awk" (respectively). Another configuration file
is seen above, "/usr/libexec/awk/addy.awk".
The big tipoff that this is a standard Linux rootkit is the compiler
inserted debugging information that includes the (edited) path to the
source file, which had better not be a valid path on Red Hat's
development system:
------------------------------------------------------------------------------
. . .
ls.c
/home/XXXXX/stuff/lrk4/fileutils-3.13/src/
gcc2_compiled.
int:t1=r1;-2147483648;2147483647;
char:t2=r2;0;127;
. . .
------------------------------------------------------------------------------
Since telnetd was not otherwise used on the system (instead SSH was
used for remote login), the intruder placed an entry (date unknown)
in /etc/inetd.conf to run in.telnetd as service "working":
------------------------------------------------------------------------------
. . .
#finger stream tcp nowait root /usr/sbin/tcpd in.fingerd
#cfinger stream tcp nowait root /usr/sbin/tcpd in.cfingerd
#systat stream tcp nowait guest /usr/sbin/tcpd /bin/ps -auwwx
#netstat stream tcp nowait guest /usr/sbin/tcpd /bin/netstat -f inet
working stream tcp nowait root /usr/sbin/tcpd in.telnetd
. . .
------------------------------------------------------------------------------
The following entry was added (date unknown) to the /etc/services file
for this service:
------------------------------------------------------------------------------
working 1120/tcp # Kerberos working daemon
------------------------------------------------------------------------------
This service port is shown in (edited) "lsof" output as listening:
------------------------------------------------------------------------------
inetd 353 root 4u IPv4 375 UDP *:talk
inetd 353 root 5u IPv4 376 UDP *:ntalk
inetd 353 root 6u IPv4 377 TCP *:working (LISTEN)
inetd 353 root 8u IPv4 378 TCP *:time (LISTEN)
inetd 353 root 10u IPv4 379 UDP *:time
inetd 353 root 11u IPv4 380 TCP *:auth (LISTEN)
inetd 353 root 12u IPv4 381 TCP *:linuxconf (LISTEN)
------------------------------------------------------------------------------
It was noted earlier that the trojan version of /bin/login allows
access using a terminal type of "vt200" (as seen by strings in the
binary), and that the mactime output shows the termcap entries for
vt200 are accessed right after the backdoor in.telnetd was accessed.
This places the last login to the system using this backdoor at
approximately 05:56 on April 15 (per the system's clock). This is
within the time window of one of the attacks logged by the network
operations engineers (Apr 15 04:11:35 to 07:32:22).
Appendix E - Analysis of handler system from Indiana University
---------------------------------------------------------------
[Note - this, too, was edited prior to publication.]
On 20 April 2000 the IT Security Office at Indiana University was
contacted by Dave Dittrich of the University of Washington (and
subsequently by Penn State). A system had been identified attempting
to use an agent similar to Trinoo to initiate denial of service
attacks using spoofed source addresses. The agent binary contained
an IP address for a host on one of IU's networks.
Our first step was to have our networks group block all traffic to
and from this IP address and log any inbound attempts. The main
intention was to prevent the machine from causing more trouble,
but we also wanted to keep the intruder from closing up shop once
our presence was known. We avoided rebooting the machine and making
physical changes to the network (e.g., unplugging the LAN cable)
in case the software running on the machine was able to detect such
events and delete itself.
Once the system was isolated from the rest of the network, we ran
the lsof program, the output of which can be found at the end of
this report.
This output indicates a process rpc.wall listening on several UDP
ports, including 6838, to which Mr. Dittrich had noted his agent
process had been sending packets. The /usr/bin/... has been opened
for reading hundreds of times. This file contained a list of IP
addresses, enciphered by a simple ASCII rotation. These IP addresses
seem to be those of agent servers controlled by this master. Indeed,
the identified agent system was on the list. We found a similar
file, /dev/grab/..., which we think might have been used by another
master server.
An example line from one of these files might look like
ckd`chj`cc`jb<
The cipher merely adds 50 to the ASCII value of each character, so the
resulting IP is
192.168.11.80
Such a file can be translated by passing it through the UNIX command
tr 'b-k`' '0-9.' | sed 's/<$//'
Using this translation, we found 76 IP addresses on the system,
presumably all DDoS slave agents.
Source code for the rpc.wall program was found by searching through
the raw disk. The attacker had done a good job of deleting the
original source file but failed to use the -pipe option to gcc.
Preprocessed code was therefore written to a temporary file to be
passed to the compiler proper. Though this temporary file was
deleted from the filesystem by the compiler, the data itself remained
intact on the disk.
Analyzing the source code of this program, we found it can perform
"stream" as well as "mstream" attacks. An mstream attack is
apparently a simultaneous stream attack on multiple IPs. We believe
this added functionality could result in larger-scale DDoS attacks
than we have seen before.
The source code found is included at the end of this report.
At this point we felt we had most of the important evidence, so we
used the dd program to copy the entire filesystem over the network
to my notebook. From there we could mount the file readonly and
examine it later. Here's what we have found so far:
ROOT KIT
A root kit called flukek.tgz was found in /bin. It contains
replacements for, among many others, cron, in.timed, inetd, login,
named, passwd, rshd, syslogd, tcpd. The following files were added
to /bin, /sbin, /usr/bin, and /usr/sbin since the system was
installed. Nothing appeared to have been added to /usr/libexec or
anything under /usr/local. Times listed are inode change times. The
file /usr/bin/old appears to be the old login binary.
-rw-r--r-- 1 root wheel 2387704 Dec 18 16:37 bin/flukek.tgz
-r-sr-xr-x 1 root daemon 12656 Apr 13 23:39 bin/login
-rw-r--r-- 1 root wheel 1401 Apr 24 12:11 usr/bin/...
-rwsr-xr-x 1 root wheel 20164 Mar 31 14:50 usr/bin/old
-rwxr-xr-x 1 root wheel 33610 Apr 13 23:37 usr/bin/rpc.wall
-rwxr-xr-x 1 root wheel 32727 Apr 5 21:56 usr/bin/xfs
-r-xr-xr-x 1 root daemon 20164 Mar 31 14:50 usr/bin/xstat
-rwxr-xr-x 1 root wheel 111500 Mar 31 20:09 usr/sbin/dnskeygen
-rwxr-xr-x 1 root wheel 266712 Mar 31 20:09 usr/sbin/irpd
-rwxr-xr-x 1 root wheel 528612 Mar 31 20:09 usr/sbin/named
-rwxr-xr-x 1 root wheel 7166 Mar 31 20:09 usr/sbin/named-bootconf
-rwxr-xr-x 1 root wheel 285076 Mar 31 20:09 usr/sbin/named-xfer
-rwxr-xr-x 1 root wheel 37056 Mar 31 20:09 usr/sbin/ndc
No libraries, kernel modules, or PAM modules appear to have been
replaced.
ACCOUNTS ADDED
The /etc/passwd and /etc/shadow files were recently modified:
-rw-r--r-- 1 root wheel 849 Feb 17 00:57 /mnt/etc/passwd
-rw------- 1 root wheel 884 Feb 17 00:57 /mnt/etc/passwd-
-r-------- 1 root wheel 794 Feb 17 00:57 /mnt/etc/shadow
-r-------- 1 root wheel 658 Nov 15 10:07 /mnt/etc/shadow-
The backup of /etc/shadow made 15 November gives us a clue what
accounts were added:
www:MyjKA0KGHplq6:11004:0:99999:7:::
login1:MyjKA0KGHplq6:11004:0:99999:7:::
web:af47L/OTL7K6.:11004:0:99999:7:::
x::11004:0:99999:7:::
I have not yet attempted to crack these passwords. I did try to
log in as "x" but was denied access.
INETD SERVICE ADDED
Inspection of /etc/services and /etc/inetd.conf reveals a service
called "a", running in.telnetd on TCP port 1111, has been added.
/etc/services:
a 1111/tcp
/etc/inetd.conf
a stream tcp nowait root /usr/sbin/tcpd in.telnetd
I haven't attempted to connect to it yet.
LOG FILES
The following three files were the only log files found to have
any pertinent information.
/.bash_history:
nslookup
cd /bin
w
ps x
ftp 192.168.0.1 21
w
/var/log/secure:
Mar 29 18:39:18 herc in.ftpd[824]: connect from 10.156.97.157
Mar 29 19:29:15 herc in.ftpd[876]: connect from 10.156.97.111
Mar 29 19:49:58 herc in.ftpd[882]: connect from 10.156.97.111
Mar 29 19:50:21 herc in.ftpd[887]: connect from 10.156.97.111
Mar 31 14:58:14 herc in.telnetd[4224]: connect from 10.54.115.105
Apr 3 23:54:02 herc in.telnetd[10403]: connect from 10.72.135.165
Apr 4 05:44:34 herc in.telnetd[11235]: connect from 10.103.26.127
Apr 4 08:28:28 herc in.ftpd[11397]: connect from 10.31.68.158
Apr 4 11:36:16 herc in.ftpd[11565]: connect from 10.31.68.158
Apr 7 05:33:32 herc in.telnetd[16737]: connect from 10.22.82.6
Apr 7 07:32:19 herc in.telnetd[16849]: connect from 10.22.82.6
Apr 7 07:33:01 herc in.telnetd[16851]: connect from 10.22.82.6
Apr 7 07:33:20 herc in.ftpd[16852]: connect from 10.22.82.6
Apr 7 07:34:11 herc in.ftpd[16855]: connect from 10.22.82.6
Apr 7 07:35:22 herc in.ftpd[16859]: connect from 10.22.82.6
Apr 7 07:37:02 herc in.rlogind[16860]: connect from 10.22.82.2
Apr 7 07:37:12 herc in.fingerd[16863]: connect from 10.22.82.2
Apr 7 07:37:18 herc in.rexecd[16866]: connect from 10.22.82.2
Apr 7 07:37:22 herc in.rshd[16867]: connect from 10.22.82.2
Apr 7 07:37:24 herc in.telnetd[16868]: connect from 10.22.82.2
Apr 7 07:37:30 herc in.ftpd[16870]: connect from 10.22.82.2
Apr 8 13:53:02 herc in.ftpd[19028]: connect from 10.247.49.53
Apr 10 23:00:05 herc in.ftpd[23304]: connect from 10.8.148.36
Apr 10 23:07:51 herc in.ftpd[23347]: connect from 10.8.148.36
Apr 13 06:50:02 herc in.telnetd[27895]: connect from 10.215.99.125
Apr 13 10:52:27 herc in.ftpd[28170]: connect from 10.114.238.145
Apr 13 10:55:50 herc in.ftpd[28171]: connect from 10.114.238.145
Apr 13 11:02:39 herc in.ftpd[28217]: connect from 10.114.238.145
Apr 16 16:29:47 herc in.ftpd[1734]: connect from 10.161.208.34
Apr 16 16:30:10 herc in.ftpd[1737]: connect from 10.161.208.34
Apr 23 18:59:36 herc in.telnetd[14746]: connect from 10.27.211.234
Apr 24 17:02:03 herc in.telnetd[16505]: connect from 10.79.16.203
/var/log/wtmp (reverse chronological order):
root pts/2 :0 Mon Apr 24 18:05 still logged in
root pts/0 :0 Mon Apr 24 17:24 still logged in
ftp ftp XXXXXX-XXXXXXXX. Thu Apr 13 10:02 - 10:02 (00:00)
ftp ftp XXXXXX-XXXXXXXX. Thu Apr 13 09:55 - 09:56 (00:00)
ftp ftp XXXXXXX-X.XXXXXX Mon Apr 10 22:07 - 22:09 (00:01)
ftp ftp XXX.XXX.82.6 Fri Apr 7 06:34 - 06:35 (00:00)
ftp ftp XXX.XXX.82.6 Fri Apr 7 06:33 - 06:34 (00:00)
ftp ftp XXXXX.XX-XXXXXXX Tue Apr 4 10:36 - 10:36 (00:00)
ftp ftp XXXXXXXX-XXXX.XX Wed Mar 29 19:50 - 19:50 (00:00)
ftp ftp XXXXXXXX-XXXX.XX Wed Mar 29 19:29 - 19:29 (00:00)
reboot system boot Wed Mar 29 16:17 (26+20:09)
OUTPUT OF lsof [edited]
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
. . .
inetd 378 root cwd DIR 3,1 1024 2 /
inetd 378 root rtd DIR 3,1 1024 2 /
inetd 378 root txt REG 3,1 18020 143469 /usr/sbin/inetd
inetd 378 root mem REG 3,1 342206 30722 /lib/ld-2.1.1.so
inetd 378 root mem REG 3,1 4016683 30729 /lib/libc-2.1.1.so
inetd 378 root mem REG 3,1 251436 30766 /lib/libnss_nisplus-2.1.1.so
inetd 378 root mem REG 3,1 364235 30742 /lib/libnsl-2.1.1.so
inetd 378 root mem REG 3,1 243964 30760 /lib/libnss_files-2.1.1.so
inetd 378 root 0u CHR 1,3 2071 /dev/null
inetd 378 root 1u CHR 1,3 2071 /dev/null
inetd 378 root 2u CHR 1,3 2071 /dev/null
inetd 378 root 4u inet 506 TCP *:ftp (LISTEN)
inetd 378 root 5u inet 507 TCP *:telnet (LISTEN)
inetd 378 root 6u inet 508 TCP *:shell (LISTEN)
inetd 378 root 7r FIFO 0,0 499 pipe
inetd 378 root 8u inet 509 TCP *:login (LISTEN)
inetd 378 root 9u CHR 5,1 2113 /dev/console
inetd 378 root 10u inet 510 TCP *:exec (LISTEN)
inetd 378 root 11u inet 511 UDP *:talk
inetd 378 root 12u inet 512 UDP *:ntalk
inetd 378 root 13u inet 513 TCP *:finger (LISTEN)
inetd 378 root 14u inet 514 TCP *:auth (LISTEN)
inetd 378 root 15u inet 515 TCP *:linuxconf (LISTEN)
inetd 378 root 16u inet 516 TCP *:a (LISTEN)
inetd 378 root 21w FIFO 0,0 499 pipe
. . .
rpc.wall 29108 root cwd DIR 3,1 31744 59393 /usr/bin
rpc.wall 29108 root rtd DIR 3,1 1024 2 /
rpc.wall 29108 root txt REG 3,1 33610 61172 /usr/bin/rpc.wall
rpc.wall 29108 root mem REG 3,1 342206 30722 /lib/ld-2.1.1.so
rpc.wall 29108 root mem REG 3,1 65996 30758 /lib/libnss_dns-2.1.1.so
rpc.wall 29108 root mem REG 3,1 4016683 30729 /lib/libc-2.1.1.so
rpc.wall 29108 root mem REG 3,1 243964 30760 /lib/libnss_files-2.1.1.so
rpc.wall 29108 root mem REG 3,1 251436 30766 /lib/libnss_nisplus-2.1.1.so
rpc.wall 29108 root mem REG 3,1 364235 30742 /lib/libnsl-2.1.1.so
rpc.wall 29108 root mem REG 3,1 251787 30764 /lib/libnss_nis-2.1.1.so
rpc.wall 29108 root mem REG 3,1 164797 30770 /lib/libresolv-2.1.1.so
rpc.wall 29108 root 0r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 1u CHR 136,1 3 /dev/pts/1
rpc.wall 29108 root 2u CHR 136,1 3 /dev/pts/1
rpc.wall 29108 root 3u inet 36066 TCP *:15104 (LISTEN)
rpc.wall 29108 root 4u CHR 136,1 3 /dev/pts/1
rpc.wall 29108 root 5u inet 36067 UDP *:6838
rpc.wall 29108 root 6r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 7r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 8r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 9r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 10r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 11r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 12r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 13r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 14r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 15r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 16r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 17r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 18r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 19r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 20r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 21u inet 38990 UDP *:2000
rpc.wall 29108 root 22r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 23u inet 38992 UDP *:2001
rpc.wall 29108 root 24r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 25u inet 38996 UDP *:2002
rpc.wall 29108 root 26r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 27u inet 38997 UDP *:2003
rpc.wall 29108 root 28r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 29u inet 39001 UDP *:2004
rpc.wall 29108 root 30r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 31u inet 39002 UDP *:2005
rpc.wall 29108 root 32r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 33u inet 39003 UDP *:2006
rpc.wall 29108 root 34r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 35u inet 39007 UDP *:2007
rpc.wall 29108 root 36r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 37u inet 39011 UDP *:2008
rpc.wall 29108 root 38r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 39u inet 39012 UDP *:2009
rpc.wall 29108 root 40r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 41u inet 39016 UDP *:2010
rpc.wall 29108 root 42r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 43u inet 39017 UDP *:2011
rpc.wall 29108 root 44r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 45u inet 39018 UDP *:2012
rpc.wall 29108 root 46r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 47u inet 39019 UDP *:2013
rpc.wall 29108 root 48r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 49u inet 39020 UDP *:2014
rpc.wall 29108 root 50r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 51u inet 39027 UDP *:2016
rpc.wall 29108 root 52r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 53u inet 39028 UDP *:2017
rpc.wall 29108 root 54r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 55u inet 39029 UDP *:2018
rpc.wall 29108 root 56r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 57u inet 39033 UDP *:2019
rpc.wall 29108 root 58r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 59u inet 39034 UDP *:2020
rpc.wall 29108 root 60r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 61u inet 39038 UDP *:2021
rpc.wall 29108 root 62r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 63u inet 39039 UDP *:2022
rpc.wall 29108 root 64r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 65r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 66u inet 39101 UDP *:2024
rpc.wall 29108 root 67r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 68r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 69u inet 39109 UDP *:2025
rpc.wall 29108 root 70r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 71u inet 39129 UDP *:2026
rpc.wall 29108 root 72r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 73u inet 39185 UDP *:2027
rpc.wall 29108 root 74r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 75u inet 39186 UDP *:2028
rpc.wall 29108 root 76r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 77u inet 39190 UDP *:2029
rpc.wall 29108 root 78r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 79u inet 39195 UDP *:2030
rpc.wall 29108 root 80r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 81u inet 39200 UDP *:2032
rpc.wall 29108 root 82r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 83u inet 39204 UDP *:2033
rpc.wall 29108 root 84r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 85u inet 39208 UDP *:2035
rpc.wall 29108 root 86r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 87u inet 39215 UDP *:2036
rpc.wall 29108 root 88r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 89u inet 39216 UDP *:2037
rpc.wall 29108 root 90r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 91u inet 39223 UDP *:2038
rpc.wall 29108 root 92r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 93r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 94r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 95r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 96r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 97r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 98r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 99r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 100r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 101r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 102r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 103r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 104r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 105r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 106r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 107r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 108r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 109r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 110r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 111r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 112r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 113u inet 39671 UDP *:2039
rpc.wall 29108 root 114r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 115r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 116r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 117r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 118r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 119r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 120r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 121r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 122r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 123r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 124r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 125r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 126r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 127r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 128r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 129r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 130r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 131r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 132r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 133r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 134r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 135r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 136u inet 41521 UDP *:2040
rpc.wall 29108 root 137r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 138r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 139r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 140r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 141u inet 41532 UDP *:2041
rpc.wall 29108 root 142r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 143r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 144u inet 41614 UDP *:2042
rpc.wall 29108 root 145r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 146u inet 41615 UDP *:2045
rpc.wall 29108 root 147r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 148u inet 41622 UDP *:2048
rpc.wall 29108 root 149r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 150u inet 41681 UDP *:2051
rpc.wall 29108 root 151r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 152u inet 41685 UDP *:2055
rpc.wall 29108 root 153r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 154u inet 41690 UDP *:2056
rpc.wall 29108 root 155r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 156u inet 41694 UDP *:2057
rpc.wall 29108 root 157r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 158u inet 41696 UDP *:2058
rpc.wall 29108 root 159r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 160u inet 41706 UDP *:2059
rpc.wall 29108 root 161r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 162u inet 41707 UDP *:2060
rpc.wall 29108 root 163r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 164u inet 41708 UDP *:2061
rpc.wall 29108 root 165r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 166u inet 41715 UDP *:2062
rpc.wall 29108 root 167r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 168u inet 41716 UDP *:2063
rpc.wall 29108 root 169r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 170r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 171r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 172r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 173r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 174r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 175r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 176r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 177r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 178r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 179r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 180r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 181r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 182r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 183r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 184r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 185r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 186r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 187r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 188r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 189r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 190r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 191r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 192r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 193r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 194r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 195r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 196r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 197r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 198r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 199r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 200r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 201r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 202r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 203r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 204r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 205r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 206r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 207r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 208r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 209r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 210r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 211r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 212r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 213r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 214r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 215r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 216r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 217r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 218r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 219r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 220r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 221r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 222r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 223r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 224r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 225r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 226r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 227r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 228r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 229r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 230r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 231r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 232r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 233r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 234r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 235r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 236r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 237r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 238r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 239r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 240r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 241r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 242r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 243r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 244r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 245r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 246r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 247r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 248r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 249r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 250r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 251r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 252r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 253r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 254r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 255r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 256r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 257r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 258r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 259r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 260r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 261r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 262r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 263r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 264r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 265r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 266r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 267r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 268r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 269r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 270r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 271r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 272r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 273r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 274r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 275r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 276r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 277r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 278r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 279r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 280r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 281r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 282r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 283r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 284u inet 53028 UDP *:2064
rpc.wall 29108 root 285r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 286u inet 53029 UDP *:2065
rpc.wall 29108 root 287r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 288u inet 53039 UDP *:2066
rpc.wall 29108 root 289r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 290u inet 53040 UDP *:2067
rpc.wall 29108 root 291r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 292u inet 53048 UDP *:2068
rpc.wall 29108 root 293r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 294u inet 53052 UDP *:2069
rpc.wall 29108 root 295r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 296r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 297u inet 53057 UDP *:2070
rpc.wall 29108 root 298r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 299u inet 53284 UDP *:2071
rpc.wall 29108 root 300r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 301u inet 53292 UDP *:2072
rpc.wall 29108 root 302r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 303r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 304u inet 53607 UDP *:2073
rpc.wall 29108 root 305r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 306u inet 53608 UDP *:2074
rpc.wall 29108 root 307r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 308u inet 53671 UDP *:2075
rpc.wall 29108 root 309r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 310u inet 53672 UDP *:2076
rpc.wall 29108 root 311r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 312u inet 53681 UDP *:2077
rpc.wall 29108 root 313r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 314u inet 53682 UDP *:2078
rpc.wall 29108 root 315r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 316u inet 53688 UDP *:2079
rpc.wall 29108 root 317r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 318u inet 53699 UDP *:2081
rpc.wall 29108 root 319r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 320u inet 53707 UDP *:2082
rpc.wall 29108 root 321r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 322u inet 53773 UDP *:2083
rpc.wall 29108 root 323r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 324u inet 53774 UDP *:2084
rpc.wall 29108 root 325r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 326u inet 53783 UDP *:2085
rpc.wall 29108 root 327r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 328u inet 53784 UDP *:2086
rpc.wall 29108 root 329r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 330u inet 53785 UDP *:2088
rpc.wall 29108 root 331r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 332u inet 53794 UDP *:2089
rpc.wall 29108 root 333r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 334u inet 53802 UDP *:2090
rpc.wall 29108 root 335r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 336u inet 53810 UDP *:2091
rpc.wall 29108 root 337r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 338u inet 53811 UDP *:2092
rpc.wall 29108 root 339r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 340r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 341u inet 54862 UDP *:2093
rpc.wall 29108 root 342r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 343r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 344r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 345r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 346r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 347r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 348u inet 55930 UDP *:2094
rpc.wall 29108 root 349r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 350u inet 56658 UDP *:2095
rpc.wall 29108 root 351r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 352r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 353r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 354u inet 57339 UDP *:2096
rpc.wall 29108 root 355r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 356r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 357u inet 57737 UDP *:2098
rpc.wall 29108 root 358r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 359u inet 57876 UDP *:2099
rpc.wall 29108 root 360r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 361u inet 57952 UDP *:2100
rpc.wall 29108 root 362r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 363r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 364r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 365u inet 58274 UDP *:zephyr-clt
rpc.wall 29108 root 366r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 367u inet 58855 UDP *:zephyr-hm
rpc.wall 29108 root 368r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 369u inet 58860 UDP *:2105
rpc.wall 29108 root 370r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 371u inet 58861 UDP *:2106
rpc.wall 29108 root 372r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 373u inet 58873 UDP *:2107
rpc.wall 29108 root 374r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 375r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 376r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 377r REG 3,1 1401 198670 /usr/bin/...
rpc.wall 29108 root 378r REG 3,1 1401 198670 /usr/bin/...
SOURCE CODE FOR rpc.wall (mstream)
[Source code removed in light of publication. See [20]]
Andrew J. Korty, Lead Security Engineer
Information Technology Security Office
Office of the Vice President for Information Technology
Indiana University
--
Dave Dittrich Client Services
dittrich@cac.washington.edu Computing & Communications
University of Washington
<a href="http://www.washington.edu/People/dad/">
Dave Dittrich / dittrich@cac.washington.edu [PGP Key]</a>
PGP 6.5.1 key fingerprint:
FE 97 0C 57 08 43 F3 EB 49 A1 0C D0 8E 0C D0 BE C8 38 CC B5
(5054580) ------------------------------------------(Ombruten)
5057051 2000-05-02 21:38 /205 rader/ Postmaster
Mottagare: Bugtraq (import) <10694>
Ärende: Re: Source code to mstream, a DDoS tool
------------------------------------------------------------
Approved-By: aleph1@SECURITYFOCUS.COM
Delivered-To: bugtraq@lists.securityfocus.com
Delivered-To: BUGTRAQ@SECURITYFOCUS.COM
X-Accept-Language: en
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <390E6E79.861F046F@moquijo.com>
Date: Tue, 2 May 2000 01:58:17 -0400
Reply-To: paul@MOQUIJO.COM
Sender: Bugtraq List <BUGTRAQ@SECURITYFOCUS.COM>
From: Paul Cardon <paul@MOQUIJO.COM>
X-To: Dave Dittrich <dittrich@CAC.WASHINGTON.EDU>
X-cc: BUGTRAQ@SECURITYFOCUS.COM, VULN-DEV@SECURITYFOCUS.COM
To: BUGTRAQ@SECURITYFOCUS.COM
Very nice writeup, especially the bits captured in the wild. I had
almost completed an analysis based on the released source only.
Instead, I will add my comments to what is a already a strong piece of
work even though Dave needed to release it sooner than anticipated. I
second the recommendation to review [12] for defense strategies.
-paul
Dave Dittrich wrote:
>
> Communication
> -------------
>
> The handler expects commands to be contained entirely in the data
> payload of a single TCP packet, not broken up character by character
> in a stream. This means that "telnet" cannot be used to control a
> handler, but instead some other client program must be used to buffer
> the command line before sending (e.g., a special command shell or port
> redirector, netcat [19], etc. -- no special client was included in the
> source posted on Security Focus [20]).
Actually, telnet can be used by switching it to line mode. This mode
is used by telnet clients by default on some OSes.
> Handler Commands
> ----------------
>
> The handler command set is:
>
> help
"commands" is an alias for "help"
> servers
> List all currently known agents.
The agents may or may not be active. This is the list of all agents
that have sent the "newserver" command to the handler at some point in
time.
> stream <hostname> <seconds>
> Begin an attack against a single host, for the specified
> duration. The handler resolves the hostname to an IP address
> and sends the command "mstream/arg1:arg1/arg2" to all agents,
> where "arg1" is the resolved hosts' IP address twice with a
> colon between (this simplifies argument parsing in the agent)
> and "arg2" is the duration in seconds.
The released source uses "mstream/arg1/arg2" without the colon and
repeated arg1. Is the colon used only in the version in the wild?
> quit
> Terminates the attacker's connection to the handler.
It also reports the termination to the connected users.
> Agent Commands
> --------------
>
> stream/IP/seconds
> Starts streaming at IP address for specified duration in
> seconds.
This does not match the description of the corresponding handler
command above but it does match the released source.
> Password protection
> -------------------
>
> (and can't simply hijack the TCP session, either, because of the command
> buffering described in the Communication section.)
Hmmm. I don't see why the session can't be hijacked.
> Fingerprints
> ------------
>
> Visible strings in the agent (in two truncated columns to save space)
> are:
>
> ------------------------------------------------------------------------------
> 131.247.208.191 _end
> 129.79.20.202 htons@@GLIBC_2.0
Did you perhaps miss these when you were sanitizing? The second one
is presumably the University of Indiana system mentioned in Appendix
E.
> Bugs in the source code for both handler and agent result in an
> increasing number of raw sockets and UDP sockets in the agent (three
> each are were witnessed on this agent), and an increasing number of
> open file handles and UDP sockets in the handler (hundreds were shown
> by Andrew Korty).
master.c neglects to do a close(fd) at the end of sendtoall and
send2server while server.c neglects to do the same at the end of
send2master.
> The stream2.c attack floods the victim with TCP ACK packets, using
> forged source addresses generated by random() (i.e., any or all
> of the four octets will occasionally be zero) and incrementing source
> port and sequence numbers, as seen in this code snippet:
By stream2.c, I assume that you mean the code sent to BUGTRAQ by Tim
Yardley that he later updated to set the ACK flag (also in [12]).
http://www.securityfocus.com/templates/archive.pike?list=1&date=2000-01-15&msg=4.2.0.58.20000121112253.012a8f10@students.uiuc.edu
The released code server.c contains DoS code ripped almost verbatim
out of stream.c with the ACK flag fix and then modified to hit
multiple targets. The only other major change is that stream.c gave
the attacker the option to select a specific destination port.
server.c always randomizes the destination port on the victim host.
If you look at Yardley's analysis (also in [12])
http://www.securityfocus.com/templates/archive.pike?list=1&date=2000-01-22&msg=4.2.0.58.20000125093641.0118b5e8@students.uiuc.edu
then you can see that the attack could have potentially worse effect
if the destination port is selected to match an open port on the
victim host.
> Weaknesses
> ----------
>
> Control communication (attacker <-> handler and handler <-> agent)
> is not encrypted, and is thus subject to session hijacking
This is a contradiction of the statement under Password Protection
above.
Other weaknesses:
The handler does not automatically clean up the file containing the
list of known agents. The stream and mstream commands will blindly
send commands to agents that no longer exist but did at one time.
The code attempts to avoid adding an ip address multiple times to the
list of known agents. However, a bug (common mistake with continue
and nearest loop) causes the check to fail in its purpose so the same
agent will have its ip address newly recorded in the agent list each
time it notifies the handler with "newserver". As a result, the
handler will send ping, stream, and mstream commands to an agent
however many times its ip address is recorded in the agent list.
In server.c note the declaration:
char *ipps[50]
and the code snippet:
if ((tmpip = strtok(ips, ":")) == NULL) continue;
ipps[0] = (char *) malloc(strlen(tmpip)+2);
strncpy(ipps[0], tmpip, strlen(tmpip)+2);
y = 1;
while ((tmpip = strtok(NULL, ":")) != NULL) {
ipps[y] = (char *)malloc(strlen(tmpip)+2);
strncpy(ipps[y], tmpip, strlen(tmpip)+2);
y++;
}
ipps[y] = NULL;
If an agent receives the mstream command containing a second argument
with greater than 50 colon-separated fields, it will merrily continue
mallocing memory and stuffing the addresses into ipps well past the
array bounds. Could get interesting when the following code attempts
to free the malloced space:
for (y = 0 ; ipps[y] != NULL ; y++) free(ipps[y]);
The list could continue but these are the major ones that affect
visible behavior. Dave was kind when he described the code as more
primitive than other known DDoS tools.
> The next logical evolutionary steps
> -----------------------------------
>
> It is not a stretch to assume that features from other published DDoS
> tools will make their way into tools like mstream. Briefly, some
> likely enhancements to this code include:
7) Target destination port selection
8) Automated cleanup of known agent list
> Appendix A - References
> -----------------------
>
> [12] BUGTRAQ threads on the stream.c DoS attack and its fallout
> http://staff.washington.edu/dittrich/misc/ddos/stream.txt
(5057051) ------------------------------------------(Ombruten)
5057579 2000-05-02 23:53 /63 rader/ Postmaster
Mottagare: Bugtraq (import) <10702>
Ärende: Re: Source code to mstream, a DDoS tool
------------------------------------------------------------
Approved-By: aleph1@SECURITYFOCUS.COM
Delivered-To: bugtraq@lists.securityfocus.com
Delivered-To: BUGTRAQ@SECURITYFOCUS.COM
X-Priority: 3
X-MSMail-Priority: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200
Message-ID: <024001bfb3ea$f612a740$6600a8c0@localarc.com>
Date: Tue, 2 May 2000 00:00:31 -0400
Reply-To: Security <security@ARC.COM>
Sender: Bugtraq List <BUGTRAQ@SECURITYFOCUS.COM>
From: Security <security@ARC.COM>
X-To: BUGTRAQ@SECURITYFOCUS.COM
To: BUGTRAQ@SECURITYFOCUS.COM
Based on the signatures provided by Mr. Dittrich, we have updated
SARA (http://www-arc.com/sara) (version 3.0.2) to detect the presence
of the mstream DDOS (both wild and published).
------------------------------------------------------------------
Bob Todd
Advanced Research Corporation
http://www-arc.com
----- Original Message -----
From: Dave Dittrich <dittrich@CAC.WASHINGTON.EDU>
To: <BUGTRAQ@SECURITYFOCUS.COM>
Sent: Monday, May 01, 2000 5:08 PM
Subject: Re: Source code to mstream, a DDoS tool
> ==========================================================================
>
> The "mstream" distributed denial of service attack tool
>
> ==========================================================================
>
> May 1, 2000
> Copyright (C) 2000. All rights reserved.
>
> David Dittrich
> University of Washington
> <dittrich@cac.washington.edu>
>
> George Weaver
> Pennsylvania State University
> <weaver@gabriel.nso.psu.edu>
>
> Sven Dietrich
> NASA Goddard Space Flight Center
> <spock@netsec.gsfc.nasa.gov>
>
> Neil Long
> Oxford University
> <neil.long@computing-services.oxford.ac.uk>
>
>
> Introduction
> ------------
>
> The following is an analysis of "mstream", a distributed denial of
> service (DDoS) attack tool, based on the source code of "stream2.c", a
> classic point-to-point DoS attack tool [12].
<<<<< cut >>>>>
(5057579) ------------------------------------------