lost and found ( for me ? )

ubuntu 12.04 64bit : install curl-loader


curl-loader is traffic generator. you can generate HTTP , HTTPS , FTP etc with curl-loader.

Add to that , you can simulate multiple clients ( multiple source IP addresses )

curl-loader

Here’s an explanation of how to install curl-loader on Ubuntu 12.04 64bit box and generate HTTP , HTTPS traffic.

# uname –r
3.2.0-39-generic

# tail -1 /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"

[ install curl-loader ]

download curl-loader from http://curl-loader.sourceforge.net/
I downloaded curl-loader-0.56.tar.bz2.

extract
# bunzip2 curl-loader-0.56.tar.bz2
# tar xvf curl-loader-0.56.tar
# cd curl-loader-0.56/

build requirements
Build it using general C development environment with bash,
gcc (3 or 4 series), make, etc on a linux machine.

Building pre-requirements are:
1. openssl binaries;
2. openssl development package with include files (on debian package libssl-dev);

I have not installed libssl-dev , so I installed libssl-dev before compiling curl-loader.
# apt-get install libssl-dev

compile
# pwd
/home/hattori/My_works/curl-loader-0.56

# make

# make install
mkdir -p /usr/bin
mkdir -p /usr/share/man/man1
mkdir -p /usr/share/man/man5
mkdir -p /usr/share/doc/curl-loader/
cp -f curl-loader /usr/bin
cp -f doc/curl-loader.1 /usr/share/man/man1/
cp -f doc/curl-loader-config.5 /usr/share/man/man5/
cp -f doc/* /usr/share/doc/curl-loader/
cp -rf conf-examples /usr/share/doc/curl-loader/

If libidns libraly name is not libidn.so , create a symbolic link file “libidn.so” like this:
# locate libidn.so
/usr/lib/x86_64-linux-gnu/libidn.so.11
/usr/lib/x86_64-linux-gnu/libidn.so.11.6.6

# cd /usr/lib/x86_64-linux-gnu/
# ln -s libidn.so.11 libidn.so

tuning
# ulimit -n 10000
# echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
# echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
# echo 102286 > /proc/sys/fs/file-max

sample configuration files are in “/usr/share/doc/curl-loader/conf-examples” directory.
# ls
10K.conf                          https.conf
60K.conf                          ipv6.conf
bax.conf                          log-hdr-body.conf
bax-login-logoff-noncycling.conf  login-uas-cycling.conf
bax-logs.conf                     login-uas-logoff-cycling.conf
<snip>

[ configuration ]

copy sample conf
# cp /usr/share/doc/curl-loader/conf-examples/bulk.conf ./my_bulk.conf

generate HTTP traffic to LVS VIP

client
 eth1 ( real ip 192.168.100.10 )
 |
vSW
 | 192.168.100.254
Router
 |
Web 192.168.200.100
  
edit my_bulk.conf  for my network environments
# cat my_bulk.conf
########### GENERAL SECTION ################################

BATCH_NAME= bulk
CLIENTS_NUM_MAX=200 # Same as CLIENTS_NUM
#CLIENTS_NUM_START=10
CLIENTS_RAMPUP_INC=5
#INTERFACE   =eth0
INTERFACE   =eth1
NETMASK=24
IP_ADDR_MIN= 192.168.100.11
IP_ADDR_MAX= 192.168.100.200
IP_SHARED_NUM=100
CYCLES_NUM= -1
URLS_NUM= 1

########### URL SECTION ####################################

#URL=http://localhost/index.html
URL=http://192.168.200.100
#URL=http://localhost/apache2-default/ACE-INSTALL.html
#URL=http://localhost/ACE-INSTALL.html
#URL_SHORT_NAME="local-apache"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 5000
TIMER_AFTER_URL_SLEEP = 500

run curl-loader
# curl-loader -f my_bulk.conf

cap data on the curl-loader box
I can simulate multiple clients with curl-loader.
# tshark -r zzz.pcap | grep -i get
Running as user "root" and group "root". This could be dangerous.
16   0.009724 192.168.100.11 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
17   0.009778 192.168.100.15 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
18   0.009815 192.168.100.14 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
19   0.009859 192.168.100.13 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
20   0.009903 192.168.100.12 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
36   0.526556 192.168.100.12 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
37   0.526639 192.168.100.13 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
38   0.526701 192.168.100.15 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
39   0.526798 192.168.100.14 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
40   0.526868 192.168.100.11 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
60   1.012545 192.168.100.16 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
65   1.012701 192.168.100.19 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
66   1.012769 192.168.100.18 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
67   1.012834 192.168.100.17 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
70   1.012958 192.168.100.20 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
86   1.051773 192.168.100.11 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
87   1.051873 192.168.100.12 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
88   1.051969 192.168.100.13 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
89   1.052063 192.168.100.14 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
90   1.052155 192.168.100.15 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1
101   1.537344 192.168.100.18 -> 192.168.200.100 HTTP 184 GET / HTTP/1.1

curl-loader logs
- bulk.log for errors and traces;
- bulk.txt for loading statistics;
- bulk.ctx for virtual client based statistics.
- bulk.ops for operational statistics.

# less bulk.log
# msec_offset cycle_no url_no client_no (ip) indic info
12 0 0 1 !! OK 200
12 0 0 4 !! OK 200
12 0 0 5 !! OK 200
12 0 0 3 !! OK 200
12 0 0 2 !! OK 200
529 1 0 4 !! OK 200
529 1 0 5 !! OK 200
<snip>

Next , generate HTTPS traffic

copy sample config
# cp /usr/share/doc/curl-loader/conf-examples/https.conf my_https.conf

edit config for my network topology.
# cat my_https.conf
########### GENERAL SECTION ################################

BATCH_NAME= https
CLIENTS_NUM_MAX=100 # Same as CLIENTS_NUM
#CLIENTS_NUM_START=1
#CLIENTS_RAMPUP_INC=1
INTERFACE   =eth1
NETMASK=24
IP_ADDR_MIN= 192.168.100.11
IP_ADDR_MAX= 192.168.100.200  #Actually - this is for self-control
CYCLES_NUM=-1
URLS_NUM= 1

########### URL SECTION ####################################

URL=https://192.168.200.100
#URL=https://localhost/apache2-default/ACE-INSTALL.html
#URL=https://localhost/ACE-INSTALL.html
#URL_SHORT_NAME="url-https"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 0 # In msec. When positive, Now it is enforced by cancelling url fetch on timeout
TIMER_AFTER_URL_SLEEP =1000

run curl-loader
# curl-loader -f my_https.conf

cap data on the curl-loader box
# tshark -r https.pcap -R '(tcp.port==443)' | grep -i syn | head -10 | grep -iv "syn, ack"
Running as user "root" and group "root". This could be dangerous.
 1   0.000000 192.168.100.11 -> 192.168.200.100 TCP 74 56998 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=20836483 TSecr=0 WS=16
 2   0.000182 192.168.100.12 -> 192.168.200.100 TCP 74 49495 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=20836483 TSecr=0 WS=16
 3   0.000357 192.168.100.13 -> 192.168.200.100 TCP 74 35063 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=20836483 TSecr=0 WS=16
 4   0.000555 192.168.100.14 -> 192.168.200.100 TCP 74 51862 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=20836484 TSecr=0 WS=16
 5   0.000724 192.168.100.15 -> 192.168.200.100 TCP 74 47794 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=20836484 TSecr=0 WS=16
 6   0.000893 192.168.100.16 -> 192.168.200.100 TCP 74 35510 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=20836484 TSecr=0 WS=16
 9   0.001116 192.168.100.17 -> 192.168.200.100 TCP 74 36015 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=20836484 TSecr=0 WS=16

IP addresses are added automatically as when starting curl-loader
# ip addr show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
   link/ether <MAC address> brd ff:ff:ff:ff:ff:ff
   inet 192.168.100.10/24 brd 192.168.100.255 scope global eth1
   inet 192.168.200.11/24 scope global eth1
   inet 192.168.200.12/24 scope global secondary eth1
   inet 192.168.200.13/24 scope global secondary eth1
   inet 192.168.100.11/24 scope global secondary eth1
   inet 192.168.100.12/24 scope global secondary eth1
   inet 192.168.100.13/24 scope global secondary eth1
   inet 192.168.100.14/24 scope global secondary eth1
<snip>

1 comment:

Note: Only a member of this blog may post a comment.