payload extraction from multiple streams - script

I had a need for a simple payload extraction from a pcap file, I wanted it to start based on my bpf expression and later extract all unique data streams that matched that expression to files in a binary format. I wanted it to be able to extract flow from almost any protocol not only TCP (for that you have tcpflow). The script is very simple it uses ngrep to do the work on the pcap file and the hex values are converted to characters using perl. There are many ways and also many tools that will extract the payload from the streams, I just needed something quick and small.

The whole script is just one loop on the output from ngrep:

my $file=shift;
my $bpf=shift;
my $ts=time();

my $conv = "";
my $buff = "";

open(CMD,"ngrep -qxlI $file \"\" \"$bpf\" |");
while(<CMD>){
  chomp;
  if (/^\S+ (\S+?):(\d+) -> (\S+?):(\d+)(\s|$)/){
    if ($conv ne "" and $buff ne ""){
      open(OUT,">> $conv");
      print OUT map(chr(hex($_)),split(" ",$buff));
      close(OUT);
    }
    $buff = "";
    $conv = "out_$1_$2_$3_$4_$ts.bin";
    print "\r.";
  }
  if (/^\s{2}(.{50})/) {
    $buff .= "$1 ";
  }
}
close(CMD);

The second regular expressions matches any hex data and appends it to the buffer. When the first regular expression is matched and if there is anything in the buffer then that data will be appended to a unique file for that conversation.

simple and easy to modify for different protocols, the full code is here.

Or you can use tshark or wireshark.

tshark -T fields -e data -r input.pcap | xxd -r -p > extracted.dat

SNMP network discovery scripts - perl, ruby and python (scapy)

Two scripts one in perl (a very old one, I'm a bit ashamed of it) one in ruby (a bit newer, logic is much better, will not hang on huge routing tables and will not kill the device with too many queries). You could be surprised how much of your network is using the default community strings.

The logic here is first to try to read some standard OIDs with a bunch of different community strings from the "seed" device. If there is a community string that works it will be moved higher in the order in which the script test them. Once the script knows the community string it will query the routing table and ARP table via SNMP to find neighbours and then repeat the whole process on them. The hosts are being distinguished by their local IP addresses, the script generates a MD5 digest from those IPs (gathered also by SNMP) to check if it is a host that was already seen.

Actually the second script was an exercise in learning/trying out ruby. It is not complete. The main function is fully implemented the missing component is a nice way of printing/storing the data.

If you would like to brute force community strings on a specific network range or a single host then I would suggest to use scapy and something like this:

>>> snmpcomm=[comm.strip() for comm in open("wordlist-common-snmp-community-strings.txt").readlines()]


>>> send(IP(dst="1.2.3.0/24")/UDP(sport=RandShort(),dport=161)/SNMP(community=snmpcomm,version=["v2c","v1","v2"],PDU=SNMPget(varbindlist=SNMPvarbind(oid="1.3.6.1.2.1.1.1.0"))))


assuming here that 1.2.3.0/24 is your target. I use send() because I just want to send the packets and I don't want to waste time on waiting for replays. In a different session on the same machine I run:

tcpdump -n udp src port 161 and net 1.2.3.0/24

and I wait for the replays. They will include the community string that triggered them.

base network adress calculation

> echo '192.168.12.43/27' | perl -pe '($_,$m)=split"/";$m*=-1;$a="";map{if(($m+=8)>0){$a.=$_>>$m<<$m}else{$a.=$_}$a.="."}split"\\.";$_=$a'

> 192.168.12.32

just so I don't forget :)

SCEP decoding/debugging

If you ever need to quickly decode SCEP messages, this is the fastest way to do it, probably.
The whole command should be on one line.

> cat request.txt | perl -pe 'use MIME::Base64;s/%([0-9a-f]{2})/sprintf("%s",pack("H2",$1))/eig;$_=MIME::Base64::decode($_);' |
openssl asn1parse -inform DER

or

> cat request.txt | perl -pe 'use MIME::Base64;s/%([0-9a-f]{2})/sprintf("%s",pack("H2",$1))/eig;$_=MIME::Base64::decode($_);' |
openssl pkcs7 -inform DER -print_certs -text

in the request.txt file is the part of the message after

"GET /scep/?operation=PKIOperation&message=" and until " HTTP/1.0"

What this is doing:

s/%([0-9a-f]{2})/sprintf("%s", pack("H2",$1))/eig <- url decoding

then (obvious):

$_=MIME::Base64::decode($_) <- Base64 decoding

and then parsing the DER:

openssl asn1parse -inform DER

or decoding pkcs7 directly:

openssl pkcs7 -inform DER -print_certs -text

An example request (for tests) can be taken from here, you can see the request in both Apache logs and the packet captures. In the above example in the request.txt file we would have (this should be one line):

> cat request.txt
MIAGCSqGSIb3DQEHAqCAMIIHiAIBATEOMAwGCCqGSIb3DQIFBQAwgAYJKoZIhvcNAQcBoIAEggJTMIAGCSqGSIb3DQEHA6CAMIICPAIBADGB%2BDCB9QIBADBeMFkxCzAJBgNVBAYTAkRFMQ8wDQYDVQQKEwZCaW50ZWMxCzAJBgNVBAsTAlJEMQ0wCwYDVQQDEwRyb290MR0wGwYJKoZIhvcNAQkBFg5yb290QGxvY2FsaG9zdAIBAzANBgkqhkiG9w0BAQEFAASBgMUB2pOa6F3APPRIAQwrIXTZn0V%2FsiTC2acRGiA%2BVUsjKsn8Hu0bMNVi8QkdHsmvoHQkJAO6z0gXKWIneh1v5q9cNkdvLmO%2BKOdRktmhOkiPRHHPzYOc5WBGhGiMQyvXkNtw5cUPO9FwdBN%2Fr%2BddRN27piYpaGJd1IG%2Fb3V8ObhyMIIBOgYJKoZIhvcNAQcBMBEGBSsOAwIHBAijOtCgU455MYCCARgNN4qTEL9xkhWwgjhyDV9NThb1l1jKa%2FJHvYjqEqiiCYRD66ho%2FukebGgXoNJe3sT3AUzZv5F7yngRoyPjpB2snCDqlRyAM0SEVxyF00o1Y6fu7gcIChSv%2BPa1sd2p6%2FIbi7UUq3JZ8u%2FsS9%2FuE8izsANx1fMqUUP6GGBsqxFKutVCk7P19dK6GvA7DajWiuJVb4WmixpwrghSjuAGNPn1ZBLW7ZdU64VlPVVJ8jppg1UrhNse68FQ4cVsXiJdqYszAcLnzqFSv11%2B%2FXxIZ9NBV02DkrGnRk6IC0CsBoez7RrOtFDRRIxNdtE0ntGJis9A70yZibkr%2Fm7ncjZy6nDgpvQjaTRVtFY5s%2B51ownh1x DyGjTPl%2FzNAAAAAAAAAACgggLsMIIC6DCCAlGgAwIBAgIRAIf%2F%2BJDqYyywkpHcqS2%2BvGEwDQYJKoZIhvcNAQEFBQAwgbExCzAJBgNVBAYTAkRFMRAwDgYDVQQIEwdHZXJtYW55MQ8wDQYDVQQKEwZCaW50ZWMxCzAJBgNVBAsTAlJEMRUwEwYDVQQDEwwxNzIuMTYuMTA0LjYxGTAXBgNVBAMTEDAwMjkwNzIwMDIwMDAyNTUxDDAKBgNVBAMTAzY3NzEQMA4GA1UEAxMHcnNhLWtleTEQMA4GA1UEAxMHY2FsaW51eDEOMAwGA1UEAxMFTGRhcDIwHhcNMDUwNDExMDgwNzQ0WhcNMDUwNDE4MDgwNzQ0WjCBsTELMAkGA1UEBhMCREUxEDAOBgNVBAgTB0dlcm1hbnkxDzANBgNVBAoTBkJpbnRlYzELMAkGA1UECxMCUkQxFTATBgNVBAMTDDE3Mi4xNi4xMDQuNjEZMBcGA1UEAxMQMDAyOTA3MjAwMjAwMDI1NTEMMAoGA1UEAxMDNjc3MRAwDgYDVQQDEwdyc2Eta2V5MRAwDgYDVQQDEwdjYWxpbnV4MQ4wDAYDVQQDEwVMZGFwMjCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA8NuMvPN%2Bf8AtK37EdgZ2ztl91LcogD4b1HD05uOtDjoHtJOgjA06wekotGh3B46XBsvrMjuDZwTPjk%2BAxm2KrbT7BdWXJwlefU7F5MP3jZFvZXXzqQCcoV8qCMCPb7WqWiPBunxSnGJBdVeFv8hrUlSeV41d8DieGkeNHne7dCcCAwEAATANBgkqhkiG9w0BAQUFAAOBgQCWClxF2ksyETsW7kF3xRy1urBL4TECGDP9ZPxv1ecMmMMH3TvBtvaumeYfQ52Bzn2B6zijzEVUW6R6%2B1SKh7zd40fVNq6GZcnFZJNwn3uXkEIwwoT2O0YUIegm8 7ChGw3IIeLP%2FSIE%2B2JEjiUzqGB9IqbVr5Q293y5OCvTUVTb%2FDGCAhcwggITAgEBMIHHMIGxMQswCQYDVQQGEwJERTEQMA4GA1UECBMHR2VybWFueTEPMA0GA1UEChMGQmludGVjMQswCQYDVQQLEwJSRDEVMBMGA1UEAxMMMTcyLjE2LjEwNC42MRkwFwYDVQQDExAwMDI5MDcyMDAyMDAwMjU1MQwwCgYDVQQDEwM2NzcxEDAOBgNVBAMTB3JzYS1rZXkxEDAOBgNVBAMTB2NhbGludXgxDjAMBgNVBAMTBUxkYXAyAhEAh%2F%2F4kOpjLLCSkdypLb68YTAMBggqhkiG9w0CBQUAoIGjMBIGCmCGSAGG%2BEUBCQIxBBMCMjAwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAfBgkqhkiG9w0BCQQxEgQQvLmfOp%2F%2FLYNUL1szSn%2BEOTAgBgpghkgBhvhFAQkFMRIEEODa6A8NT%2FkEvkFZ6eclspcwMAYKYIZIAYb4RQEJBzEiEyA4N0ZGRjg5MEVBNjMyQ0IwOTI5MURDQTkyREJFQkM2MTANBgkqhkiG9w0BAQEFAASBgLFndLx8XykdWU7QevIbZyBrTOnS3pGeW1s0sC786r9%2BDbe7LNWPJlDnL1FUmcavFYFIrxP4O4rSxCrLne7TSmfQeJMDSD63TWNt5Ou%2BpwhRQf8PZLeXLzITjg6yQY%2FgVc%2Flf7XIGO%2FUnJHr7q%2FPd%2B6gQAr%2FxTFgpq8WlsfN7s8UAAAAAA%3D%3D

You can download the file used in those examples from here.
The truncated output would look like this (-i in asn1parse added for nicer printing):

> cat request.txt| perl -pe 'use MIME::Base64;s/%([0-9a-f]{2})/sprintf("%s",pack("H2",$1))/eig;$_=MIME::Base64::decode($_);' | openssl asn1parse -i -inform DER | head -n 20 

    0:d=0  hl=2 l=inf  cons: SEQUENCE          
    2:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-signedData
   13:d=1  hl=2 l=inf  cons:  cont [ 0 ]        
   15:d=2  hl=4 l=1928 cons:   SEQUENCE          
   19:d=3  hl=2 l=   1 prim:    INTEGER           :01
   22:d=3  hl=2 l=  14 cons:    SET               
   24:d=4  hl=2 l=  12 cons:     SEQUENCE          
   26:d=5  hl=2 l=   8 prim:      OBJECT            :md5
   36:d=5  hl=2 l=   0 prim:      NULL              
   38:d=3  hl=2 l=inf  cons:    SEQUENCE          
   40:d=4  hl=2 l=   9 prim:     OBJECT            :pkcs7-data
   51:d=4  hl=2 l=inf  cons:     cont [ 0 ]        
   53:d=5  hl=4 l= 595 prim:      OCTET STRING      [HEX DUMP]:308006092A864886F70D010703A0803082023C0201003181F83081F5020100305E3059310B3009060355040613024445310F300D060355040A130642696E746563310B3009060355040B13025244310D300B06035504031304726F6F74311D301B06092A864886F70D010901160E726F6F74406C6F63616C686F7374020103300D06092A864886F70D0101010500048180C501DA939AE85DC03CF448010C2B2174D99F457FB224C2D9A7111A203E554B232AC9FC1EED1B30D562F1091D1EC9AFA074242403BACF48172962277A1D6FE6AF5C36476F2E63BE28E75192D9A13A488F4471CFCD839CE5604684688C432BD790DB70E5C50F3BD17074137FAFE75D44DDBBA6262968625DD481BF6F757C39B8723082013A06092A864886F70D010701301106052B0E0302070408A33AD0A0538E7931808201180D378A9310BF719215B08238720D5F4D4E16F59758CA6BF247BD88EA12A8A2098443EBA868FEE91E6C6817A0D25EDEC4F7014CD9BF917BCA7811A323E3A41DAC9C20EA951C80334484571C85D34A3563A7EEEE07080A14AFF8F6B5B1DDA9EBF21B8BB514AB7259F2EFEC4BDFEE13C8B3B00371D5F32A5143FA18606CAB114ABAD54293B3F5F5D2BA1AF03B0DA8D68AE2556F85A68B1A70AE08528EE00634F9F56412D6ED9754EB85653D5549F23A6983552B84DB1EEBC150E1C56C5E225DA98B3301C2E7CEA152BF5D7EFD7C4867D341574D8392B1A7464E880B40AC0687B3ED1ACEB450D1448C4D76D1349ED1898ACF40EF4C9989B92BFE6EE7723672EA70E0A6F423693455B45639B3EE75A309E1D710F21A34CF97FCCD00000000
  652:d=5  hl=2 l=   0 prim:      EOC               
  654:d=4  hl=2 l=   0 prim:     EOC               
  656:d=3  hl=4 l= 748 cons:    cont [ 0 ]        
  660:d=4  hl=4 l= 744 cons:     SEQUENCE          
  664:d=5  hl=4 l= 593 cons:      SEQUENCE          
  668:d=6  hl=2 l=   3 cons:       cont [ 0 ]        
  670:d=7  hl=2 l=   1 prim:        INTEGER           :02

As suggested in the comments using as the last command instead of "openssl asn1parse ..." the command "openssl pkcs7 -inform DER -print_certs -text" gives an even nicer looking output (the same data as previously):

> cat request.txt |\ 
 perl -pe 'use MIME::Base64;s/%([0-9a-f]{2})/sprintf("%s",pack("H2",$1))/eig;$_=MIME::Base64::decode($_);' |\
 openssl pkcs7 -inform DER -print_certs -text
Certificate:
Data:
        Version: 3 (0x2)
        Serial Number:
            87:ff:f8:90:ea:63:2c:b0:92:91:dc:a9:2d:be:bc:61
Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=DE, ST=Germany, O=Bintec, OU=RD, CN=172.16.104.6, CN=0029072002000255, CN=677, CN=rsa-key, CN=calinux, CN=Ldap2
        Validity
            Not Before: Apr 11 08:07:44 2005 GMT
            Not After : Apr 18 08:07:44 2005 GMT
        Subject: C=DE, ST=Germany, O=Bintec, OU=RD, CN=172.16.104.6, CN=0029072002000255, CN=677, CN=rsa-key, CN=calinux, CN=Ldap2
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:f0:db:8c:bc:f3:7e:7f:c0:2d:2b:7e:c4:76:06:
                    76:ce:d9:7d:d4:b7:28:80:3e:1b:d4:70:f4:e6:e3:
                    ad:0e:3a:07:b4:93:a0:8c:0d:3a:c1:e9:28:b4:68:
                    77:07:8e:97:06:cb:eb:32:3b:83:67:04:cf:8e:4f:
                    80:c6:6d:8a:ad:b4:fb:05:d5:97:27:09:5e:7d:4e:
                    c5:e4:c3:f7:8d:91:6f:65:75:f3:a9:00:9c:a1:5f:
                    2a:08:c0:8f:6f:b5:aa:5a:23:c1:ba:7c:52:9c:62:
                    41:75:57:85:bf:c8:6b:52:54:9e:57:8d:5d:f0:38:
                    9e:1a:47:8d:1e:77:bb:74:27
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha1WithRSAEncryption
         96:0a:5c:45:da:4b:32:11:3b:16:ee:41:77:c5:1c:b5:ba:b0:
         4b:e1:31:02:18:33:fd:64:fc:6f:d5:e7:0c:98:c3:07:dd:3b:
         c1:b6:f6:ae:99:e6:1f:43:9d:81:ce:7d:81:eb:38:a3:cc:45:
         54:5b:a4:7a:fb:54:8a:87:bc:dd:e3:47:d5:36:ae:86:65:c9:
         c5:64:93:70:9f:7b:97:90:42:30:c2:84:f6:3b:46:14:21:e8:
         26:f3:b0:a1:1b:0d:c8:21:e2:cf:fd:22:04:fb:62:44:8e:25:
         33:a8:60:7d:22:a6:d5:af:94:36:f7:7c:b9:38:2b:d3:51:54:
         db:fc
-----BEGIN CERTIFICATE-----
MIIC6DCCAlGgAwIBAgIRAIf/+JDqYyywkpHcqS2+vGEwDQYJKoZIhvcNAQEFBQAw
gbExCzAJBgNVBAYTAkRFMRAwDgYDVQQIEwdHZXJtYW55MQ8wDQYDVQQKEwZCaW50
ZWMxCzAJBgNVBAsTAlJEMRUwEwYDVQQDEwwxNzIuMTYuMTA0LjYxGTAXBgNVBAMT
EDAwMjkwNzIwMDIwMDAyNTUxDDAKBgNVBAMTAzY3NzEQMA4GA1UEAxMHcnNhLWtl
eTEQMA4GA1UEAxMHY2FsaW51eDEOMAwGA1UEAxMFTGRhcDIwHhcNMDUwNDExMDgw
NzQ0WhcNMDUwNDE4MDgwNzQ0WjCBsTELMAkGA1UEBhMCREUxEDAOBgNVBAgTB0dl
cm1hbnkxDzANBgNVBAoTBkJpbnRlYzELMAkGA1UECxMCUkQxFTATBgNVBAMTDDE3
Mi4xNi4xMDQuNjEZMBcGA1UEAxMQMDAyOTA3MjAwMjAwMDI1NTEMMAoGA1UEAxMD
Njc3MRAwDgYDVQQDEwdyc2Eta2V5MRAwDgYDVQQDEwdjYWxpbnV4MQ4wDAYDVQQD
EwVMZGFwMjCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA8NuMvPN+f8AtK37E
dgZ2ztl91LcogD4b1HD05uOtDjoHtJOgjA06wekotGh3B46XBsvrMjuDZwTPjk+A
xm2KrbT7BdWXJwlefU7F5MP3jZFvZXXzqQCcoV8qCMCPb7WqWiPBunxSnGJBdVeF
v8hrUlSeV41d8DieGkeNHne7dCcCAwEAATANBgkqhkiG9w0BAQUFAAOBgQCWClxF
2ksyETsW7kF3xRy1urBL4TECGDP9ZPxv1ecMmMMH3TvBtvaumeYfQ52Bzn2B6zij
zEVUW6R6+1SKh7zd40fVNq6GZcnFZJNwn3uXkEIwwoT2O0YUIegm87ChGw3IIeLP
/SIE+2JEjiUzqGB9IqbVr5Q293y5OCvTUVTb/A==
-----END CERTIFICATE-----

getting to the IP data encapsulated inside some strange protocol

An example of striping outside encapsulation from the 

packets to get to the IPv4 header inside the packet:

tcpdump -nxr input.pcap | \

perl -e '$i=0;while(<>){if (/^\d/){ $i=0 }; \

if (/0x0000: /) { $i=1;print "\n";}; \

if($i==1){s/^.+?: //;s/(..)(..)\s/$1 $2 /g;print $_;}}' \

| perl -npe 's/^.+? 45 /00000 45 /' | \

| text2pcap -e 0x800 - output.pcap

Nothing ground breaking but could be useful,

or a slightly shorter version (added 2012-10-25):

tcpdump -nxr input.pcap |\

perl -ne 'next if (not /^\s+0x0/);chomp;s/^/\n/ if (/0x0000/);\

s/\t+0x.+?:\s+//;s/\ //g;print' |\

perl -pe 's/^.+?4500/4500/;s/(..)/$1 /g;s/^/000000 /' |\
text2pcap -e 0x800 - output.pcap