The CBL - Composite Blocking List

How to find BOTs in a LAN

Table of Contents

THIS DOCUMENT IS UNDER DEVELOPMENT. Comments are welcome.

Introduction

Many times people have a CBL listing that corresponds to the NAT or PAT for a LAN, and it can be EXTREMELY difficult identifying the infected machine. This page mentions a number of simple-to-advanced methods for identifying infected machines on a LAN. Many are methods that non-technical people may well not understand nor be able to conveniently implement.

Your first line of defence if you use a NAT or PAT firewall is to make sure that your NAT does not allow inbound or outbound port 25 connections _except_ to your mail server (if you have one).

But whether or not your NAT is secured, you will still need to be able to find the infected machine.

This page is intended for a broad range of levels of experience. Both network neophytes and experts should be able to find useful tidbits of information in it.

Some of these methods are relatively easy for anyone to use, so we'll mention them with brief discussions on how to use them.

Other methods really aren't suitable for network neophytes. We mention them in passing so that if you are capable of doing them, or can hire a consultant who can, you/they will know what to look for.

These methods are mostly independent of what kinds of computers or operating systems you're using. The tool names may change between, say, Linux and Windows, but you're looking for the same things. In the discussions below, we'll refer to UNIX/Linux/MacOS/FreeBSD/Solaris/AIX/NetBSD/etc as "*NIX".

The author has taken a stab at identifying which methods are easy, moderately difficult or hard by something like this: "[HARD]".

Within each of the two sections (per-machine and centralized) we present them in an "easy to harder" order. Review them in order to find out which will be the most appropriate for you to use.

But before you try to find out what machine it is, SECURE YOUR NAT.

What will Anti-Virus (A/V) software do for me?

Basically, not much.

These days most bot infections cannot be found by anti-virus "cleaners", or at least not without having to try a dozen or more of them. This means you can expend a considerable amount of time and effort running your A/V tools on every machine on your LAN and find absolutely nothing. Or find something that has nothing whatsoever to do with the CBL listing.

For another voice on current A/V effectiveness see Gary Warner's blog. One of the additional things that Gary omitted mentioning is that of "polymorphic viruses". Signature-based A/V works by taking a MD5 hash (a checksum) of the malicious program, and saving the hash as the "signature". Then, whenever anyone else sees a file with the same MD5 hash, they know its the same file, and hence the same malware. Problem is that there are an infinite number of ways that an executable program can be "packed" on disk. In the old days, the virus would be packed once, and distributed that way. These days, the virus downloaders have the capability of changing the packing every time the file is downloaded. Meaning you'd need an infinite number of MD5 hashes to catch it. And if you've not seen that particular packing before (you may be the only person who'll ever get that packing), then, you won't have an MD5 hash for it.

There's another breed of virus scanners which "decode" the program and try to figure out what it's going to do - "behavioral detection". These aren't very good yet, and they're very very slow. But we're hoping they'll get there.

As a consequence of all this, even if you did know which machine was infected, the A/V tools wouldn't fix them.

The result basically being that A/V tools can't be used to find which machine is infected, and even if you did know which machine was infected, you can't successfully clean it, you have to reinstall your software.

Ugly? Yes. Frustrating? Yes. But that's how things are now.

This document focusses on how to find the infected machine. Once you have found it, you generally will have to reinstall it.

What am I looking for?

The essential goal of this exercise is to figure out which computer is infected and sending email.

The methods we describe here are how to find out which machine is sending lots of email. You will be looking for direct evidence that a particular computer is sending email it shouldn't be, OR, indirect evidence that it is doing it.

Particularly in a large network (with 100s or 1000s of computers) you will want a "central detection" method. In other words, you look in one place, and it tells you which computer is sending lots of email.

We discuss a number of methods under the "Centralized Detection" section below, however, many of these require significant network monitoring/admin expertise and/or testing hardware. Environments run by professional network engineers are frequently able to do these as a normal course of events, but this is seldom the case in a small business or home network. Though, a small business should be able to hire a consultant who could use some of these methods.

The simplest methods under Centralized Detection are using a network sniffer or firewall logging.

Depending on how your network is set up, a network sniffer won't work without considerable extra effort. This is because modern higher performance networking gear makes network sniffing quite difficult.

If your LAN uses an ethernet hub (not a network switch or router), OR, your firewall IS a generalized computer (eg: Linux or Windows server acting as a firewall) go directly to the port 25 sniffing section below. If you're not using a hub, sniffing is still possible, but it's harder, and using one of the per-machine methods may be simpler.

If you have a decent firewall that has logging capabilities, go to the section on Firewall logging.

What am I NOT looking for?

CBL listing criteria is very narrow:

Per-machine methods

The methods in this section require that you check out each computer in your LAN individually.

If you have a number of machines to check, particularly windows machines, we recommend downloading some of the tools we mention (or others you may find) and put them on a USB key. Then you can go from machine to machine, plugging in the USB key, and running each of the tools without too much difficulty. We recommend trying the tools mentioned here before spending lots of time with A/V scanners.

tcpview/tcpvcon (Windows) [EASY]

tcpview and tcpvcon are free and can be obtained from Microsoft.

If you don't want to download anything, you can use Windows netstat (see the next section) instead. tcpview's display makes it a bit easier to find viruses, but, basically netstat is the same thing. This seems to be standard on Windows.

tcpview and tcpvcon are windows and and command-line based versions of something similar to *NIX netstat. These are good tools to have on a USB key "toolkit".

Navigate to where you've placed tcpview (perhaps on a USB key), and run it. It will display all of the programs that have network connections open - naming the program, protocol, local address and port, remote address and port and state.

You're looking for lines that have the remote address say ":smtp" or ":25", representing a remote email connection. A machine should not have any of these except when it's actively sending email. On an end-user desktop, there shouldn't be any at all unless the user is sending an email at the time.

It's often a good idea to shut down the user's mail reader and other unnecessary programs (like browsers etc) when you're doing this so you don't get confused with a flood of irrelevant information. However, some BOTs actually run inside mail readers (especially Outlook), so you should try first with the mail reader shut down, and if you don't find anything, start it up again and watch some more.

When a connection is freshly established, the corresponding line is green. When the connection ends, it's shown in red briefly before disppearing.

If you have found the machine with a high volume bot, which could be sending dozens or hundreds of emails per minute, the display will light up like a christmas tree with large numbers of green ":smtp" lines appearing and red ":smtp" lines disappearing very rapidly. The bot may be deliberately slow, and only send emails sporadically. So watch the display for a few minutes to see if any ":smtp" lines show up and disappear.

If you find the machine with the bot showing up on tcpview, the temptation is strong to simply delete the corresponding program. Don't. Chances are high that it's an infection inside a legitimate windows program - deleting it will cripple the machine, or, that there is software in place to replace it after you have removed it.

Instead, obtain and run as many anti-virus programs as you can, and see if any detect or remove it. After this, reboot the machine, and run tcpview again. Watch it for a while. If the problem recurs, you will have to reinstall the computer from scratch.

Note: There are a few bots this won't work with - Srizbi and Xarvester have their own TCP stacks, and it's believed that tcpview won't see their activity.

Netstat (*NIX and Windows) [EASY-MEDIUM]

Netstat is similar in intent to the tcpvcon version of tcpview, and is standard on most versions of *NIX - it's been around for decades. Secondly, most versions of Windows have it. The main difference with tcpview is that netstat is a command line function that takes a single snapshot of current connections.

In many versions of netstat, the most effective command line to use is: 

netstat -nap

Which could, in the case of Darkmailer, show an active infection like this: 

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      1 192.168.2.2:58246       212.69.102.240:25       SYN_SENT    12614/b.pl
tcp        0      0 192.168.2.2:35843       209.85.201.27:25        ESTABLISHED 7996/ciwhcnsb.pl
tcp        0      0 192.168.2.2:53051       81.13.48.2:25           TIME_WAIT   -
tcp        0      0 192.168.2.2:53623       77.243.121.126:25       TIME_WAIT   -
tcp        0      0 192.168.2.2:57816       217.13.210.81:25        TIME_WAIT   -
tcp        0      1 192.168.2.2:50531       217.16.16.81:25         SYN_SENT    12270/nxhbo.pl
tcp        0      0 192.168.2.2:52437       217.198.11.26:25        TIME_WAIT   -
tcp        0      1 192.168.2.2:50140       195.64.222.2:25         SYN_SENT    9273/yzezihd.pl

The ":25" under "Foreign Address" indicates an outbound SMTP connection. "NNNN/name" under "PID/Progran name" is the process id and process name of the offending program. The large variety of "states" show that it's starting up/shutting down connections very quickly.

Most if not all versions of Windows have a "netstat" DOS command. One corporate security person once said "I haven't yet had netstat fail to find an infected machine".

On Windows, use this in a dos command window: 

netstat 5
This will give you a list of all network connections your machine has open, much like *NIX netstat above every 5 seconds until you stop it. You're looking for very much the same sort of things as *NIX netstat above. You'll probably see Microsoft, Yahoo and other familiar names - they're normal (from your browser, IM etc). "Akamai" perhaps won't be familiar, but it's normal too. Lots of port 25 connections is the usual sign of infection.

In *NIX etc, it's often enough to find the listed programs and remove it, tho, that will not necessarily prevent you from being infected again. Especially with Darkmailer. Also with Darkmailer, you often won't be able to find the programs, because they start up, delete themselves from the file system, and continue running in memory.

Note: you will usually see a lot more lines than the above that do not have ":25", those are other non-email connections. You might want to repeatedly pipe the output of "netstat -nap" through "grep :25" to only see the SMTP connections. ":25" on the local address means an inbound connection.

"New files" in System Directories (Windows) [EASY]

Most infections "drop" their malicious programs in Windows "system" directories. For example, the interesting directories on Windows/XP are C:\windows\system and C:\windows\system32.

It's often possible to see these programs by navigating to the system directories, switching to the "detailed view" and then sorting by date. There is a good chance that the malicious software on your machine was created within the past 30 days. If you find programs in these directories that are that young, and aren't explainable by a new software install or patch:

Other Tools (Windows, per-machine)

There are a variety of other tools you can use on a per-machine basis, but these are generally considerably more effort if you have a lot of computers to check.

Important note: "full and most effective" use of seccheck and hijackthis means that you're asking other people to provide you with free support. A good analysis could take quite a while - that's a lot to ask of someone. By all means use these tools on any/all of your machines, but please only ask for analysis assistance on the one or few machine[s] that appear suspicious. In other words, don't send dozens or hundreds of reports for expert analysis. It isn't fair to them.

Centralized Detection

Firewall logging [EASY-HARD]

Many firewalls can be configured to log outbound port 25 connections. If your firewall is logging such connections, you can usually identify very quickly the offending machine by lots of "mysterious" outbound port 25 connections. Eg: non-mail servers making dozens or hundreds of outbound port 25 connections per minute.

Obviously, if your firewall doesn't support this kind of logging (many inexpensive consumers-grade firewalls can't), this becomes pretty hard to do.

Firewalls and UPNP

Universal Plug and Play (UPNP) is a feature of many routers and gateways (particularly consumer equipment) that permits computers on the local LAN to reconfigure the router. These are usually used by online games, certain VOIP hardware and other things.

Certain spambots (Rustock in particular) use UPNP commands to subvert port 25 blocking.

If your router/NAT supports UPNP, check to ensure that UPNP logging is turned on. You should be able to see log records showing internal computers making UPNP changes. Unless they're legitimate applications, it's probably a spambot, likely Rustock. Hopefully the log may show you the IP address of the infected machine.

Make sure that UPNP is disabled unless you absolutely need it.

Port 25 sniffing [EASY-HARD]

This is listed as "EASY" if you have a hub-based network. It gets harder if you don't.

Port 25 sniffing is a powerful network diagnostic tool that when used correctly can find just about any malicious machine or program. They work by running a program on one of your machines with network set to "promiscuous mode", which allows it to see and analyze all network traffic on your LAN. Just look for lots of port 25 connections coming from machines that shouldn't be sending any or much email.

There are hardware and software sniffers available. Hardware sniffers are fairly specialized equipment, and are often too expensive for purposes like this. Software sniffers are usually more practical. The most popular and powerful software sniffer freely available is Wireshark, which runs on *NIX, Windows and other systems.

For a howto guide of how to use Wireshark, see MyNetwatchman

Please read the discussion on how to set up a sniffer. Note in particular, item 4 - "switched Ethernet" - most networks are set up with switches these days, and it makes it difficult to get sniffers to listen to the whole network. In section 4, think of "host A" as the infected computer (you don't know what it is), and "Host B" is the NAT.

The problem is that most relatively modern LAN networks are based upon "proper" routers or network switches. What this means is that each wire from the switch to a given computer only carries the traffic for the IP corresponding to that computer. Not the rest of the LAN. Hence, the sniffer sitting on a switched port only sees traffic to the sniffer machine - useless.

With a hub, it doesn't know which wire is which, and sends a copy of all packets down each port, so a sniffer on one of the ports can see all traffic traversing the hub.

If your computers are connected together with hubs, it's easy, install wireshark on one of the computers "near" the NAT and just start sniffing.

If you're unsure as to whether the sniffer is going to work for you in your network, install wireshark, and from another computer, do "something" to the Internet. If you can't see the network traffic in the sniffer, you probably have a switched network. [If you're unsure of what to look for, install tcpview on a machine and see what Internet connections it has open. The sniffer should be able to "see" those connections on the wire.]

In a switched network, you somehow have to get a non-switched drop (for the sniffer machine) connected to the LAN segment that talks to your external Internet connection. Eg: on the wire between the NAT device (perhaps a discrete firewall or your ADSL modem) and the rest of your LAN. It must be on the LAN side of the NAT.

If you're lucky, you have a "monitoring" or "mirror port" on your switch, or some other way of making one of the switch ports open. Just attach the sniffer machine there.

Without a monitor port, another way of solving this is to find a "ethernet hub". Which is a simple device with several RJ45 network connectors, and often doesn't even have a power supply. You can often find these in computer stores used parts bins, and even brand new one should cost less than $20. Just be sure it's an "ethernet hub", not an "ethernet switch". If in doubt, ask the salesperson. If the salesperson doesn't know, check the Internet. Connect the hub between your NAT and the rest of the network, then connect your sniffer machine to one of the other hub ports. This is the "hubbing out" diagram.

Note that if your NAT gateway is an integrated firewall/router this can be problemmatic. In such cases, you'll have to rely on firewall rules and logs instead of a sniffer, or add a cheap switch (1Gb switches are < $40) for all of your computers. From the switch, you run a line to the hub, and from the hub to your firewall/router, with the sniffer hanging off one of the hub ports.

[This author has a Dlink wireless 4 port router (100Mb) implementing the NAT connection to the Internet modem and wireless connections. Only one of the Dlink's LAN ports is used - it connects to a 1000Mb switch, where all the wired computers connect to. This was cheaper than upgrading the wireless router to allow the higher speed wired machines to talk at 1000Mb. If a sniffer was necessary, it would be connected via an old 10Mb passive hub between the switch and the router - no particular performance penalty, because essentially the only traffic going through this link is to the Internet, and the author's Internet connection isn't that fast. This doesn't necessarily help sniff the wireless connections, however, machines could be moved to wired connections for testing.]

There are network sniffers that can trick switches into behaving like hubs. Eg: the "dsniff" sniffer - see the Capture using a MITM (Man-In-The-Middle) software for more detail. Take special note of the warnings - use with caution.

Command and Control Detection [MODERATE-HARD]

Spambots are controlled by criminals (botmasters) in a variety of different ways, which can be differentiated in the following ways based on who connects to what, and how they can be detected:

The methods we describe below won't always be terribly useful, but if you have a sniffer working properly, they won't take much time to try, and you may get lucky.

Odd DNS MX query sources [MODERATE-HARD]

To send email, virtually all BOTs have to issue DNS MX queries to find how to deliver their spam/viruses. Under normal circumstances, ONLY your mail server[s], your DNS server[s] (if any) should be issuing MX queries. Web servers that do direct-to-recipient emailing will do MX queries too, but this is generally unwise, and you should force your web server's email through your main mail server.

End user computers generally do NOT have to issue MX queries - they just hand the email off to your mail server (by explicit "smarthost", "relay", "smtp server" or "outbound mail server" settings), which will require an A record but not MX record lookup).

This means that a BOT sending lots of spam will do lots of MX queries. If you find a end-user computer or some other computer that shouldn't be doing email at all doing MX queries (especially lots of them), you've found the infected computer[s].

If you have your own DNS server (eg: a DNS cache), you should be able to get the DNS server to give you basic statistics of who is issuing MX queries to them. If you don't have your own DNS server, you could look for unusual sources of DNS MX queries via a sniffer. It may be easier to sniff all DNS traffic going to your DNS server than your firewall.

Note some BOTs undoubtably use their own DNS servers, and ignore your local settings. In which case, you'd have to be able to sniff all Internet-bound traffic looking for DNS traffic not coming from your DNS cache.

Note: it's probably a good idea to configure your firewall to only allow your DNS cache to send/receive DNS packets (UDP port 53) to/from the Internet. This has a number of benefits, including disabling some bots, and completely disrupting DNS hijacking attacks, which are becoming a major hazard on the Internet (phishing, man-in-the-middle bank account attacks etc). This is fairly easy to do if you allocate most IPs via DHCP, but you will have to remember to check the DNS server settings on your static IP computers. However, in large ad-hoc networks this may be impractical to implement - too much work fixing computer DNS settings.

Configuring DNS servers to yield detailed per-IP metrics is beyond the scope of this page. See the documentation for your DNS server. However, a team member provided this configuration snippet on how to make BIND log queries: 

logging {
        channel "logger" {
                file "/var/log/named.log" versions 3 size 5m;
                severity debug 5;
                print-time yes;
                print-category yes;
        };
        category queries {
                 "logger";
        };
};

This will log all (not just MX) queries in /var/log/named.log, example: 

28-Jul-2009 15:07:09.206 queries: client 192.168.13.100#59889: query: somebody.com IN MX +

You could then issue "grep MX /var/log/named.log" and see if some unexpected IP address is doing a lot of MX queries. See BIND for more information on logging options.

Lots of DNS NXDOMAINs [MODERATE-HARD]

Some BOTs (eg: Conficker) use DNS to periodically find their command-and-control (C&C) servers. In some cases, some of the C&Cs have been taken down, or, the BOT uses time-based algorithms to compute the names before the domain name is registered.

As a consequence such BOTS will do DNS A record queries in bursts, and often get a lot of "no such name" (NXDOMAIN) responses. Lots of "NXDOMAIN" isn't normal behaviour, particularly for end-user computers. This can most often be found if you have your own DNS server - see previous section about setting up logging.

Port Scanners [EASY-MODERATE]

Back in the days before "outbound controlled BOTs", port scanners were frequently used to scan your own computers to see what ports are open. In this way you could often find the port on which the BOT was listening, or determine that the computer was offering services it didn't need to, and turn them off.

Port scanners are of relatively little use with more modern spambots - the infection is not listening for inbound C&C connections, it makes the connections itself outbound. Secondly, with NATs, the C&C server couldn't reach the infected computer anyway.

However, sometimes you get lucky. Some bots have provisions for multiple C&C methods, or install open proxies or..., these a port scanner can find.

The most common/popular port scanner is the venerable Nmap tool. It's free, and runs on just about anything.

But first, two warnings:

Detailed description of how to use nmap is well beyond the scope of this paper. For our purposes, the following command will do most of what you want and be non-destructive - won't do any damage: 

nmap -A [machine or network specification]

For machine specification, you can just give the machine's name. To scan an entire network, say, all of 192.168.0.0-192.168.0.255, use "192.168.0.0/24".

The above command will show what ports are open (and thus listening), and usually what they're used for. Most machines should only be listening on a few. Listening on ports >1024 and/or that don't have a "name" under SERVICE are suspicious and require closer looking at. Machines that shouldn't have a web server listening on ports 80, 8080 are worth looking at. Machines that shouldn't be running services at all should be looked at. tcpview or "netstat -nap" can be used on the machine to find out what's listening on that port.

End Notes