CentOS6.5: monitor bandwidth with vnstat

# cat /etc/centos-release ;uname -ri CentOS release 6.5 (Final) 2.6.32-431.3.1.el6.x86_64 x86_64

Add epel repo

# cat /etc/yum.repos.d/epel.repo [epel] name=Extra Packages for Enterprise Linux 6 - $basearch #baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch failovermethod=priority enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 [epel-debuginfo] name=Extra Packages for Enterprise Linux 6 - $basearch - Debug #baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch/debug mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-6&arch=$basearch failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 gpgcheck=1 [epel-source] name=Extra Packages for Enterprise Linux 6 - $basearch - Source #baseurl=http://download.fedoraproject.org/pub/epel/6/SRPMS mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-6&arch=$basearch failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 gpgcheck=1

Install vnstat via yum

# yum update -y # yum install vnstat –y

config file

# less /etc/vnstat.conf # vnStat 1.11 config file ## # default interface #Interface "eth0" Interface "br0" # location of the database directory DatabaseDir "/var/lib/vnstat"

run vnstat as daemon.

# /etc/init.d/vnstat start vnstatd を起動中:                                          [  OK  ]

# vnstat Database updated: Tue Jan 14 15:33:21 2014   br0 since 01/14/14          rx:  10.46 MiB      tx:  14.21 MiB      total:  24.67 MiB   monthly                     rx      |     tx      |    total    |   avg. rate     ------------------------+-------------+-------------+---------------       Jan '14     10.46 MiB |   14.21 MiB |   24.67 MiB |    0.17 kbit/s     ------------------------+-------------+-------------+---------------     estimated        22 MiB |      31 MiB |      53 MiB |   daily                     rx      |     tx      |    total    |   avg. rate     ------------------------+-------------+-------------+---------------         today     10.46 MiB |   14.21 MiB |   24.67 MiB |    3.61 kbit/s     ------------------------+-------------+-------------+---------------     estimated        15 MiB |      21 MiB |      36 MiB |

dump bandwidth as graph

# vnstati -i br0 -vs -o hourly.png

It seems that vnstat can not output bandwidth consumption per port or IP addresses.

Anyway, this tool is easy to use to monitor total bandwidth.

how to install / use libvirt-snmp on CentOS6

libvirt-snmp allows you to get KVM-related info via snmpget , snmptrap.
Add to that , you can shutdown , pause , start VMs via snmpset.
Here’s an explanation of how to install / use libvirt-snmp.

I just referred to http://wiki.libvirt.org/page/Libvirt-snmp#libvirt-snmp.
many thanks!

# uname –ri 2.6.32-279.19.1.el6.x86_64 x86_64 # cat /etc/centos-release CentOS release 6.3 (Final) # /usr/libexec/qemu-kvm --version QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard # libvirtd --version libvirtd (libvirt) 0.9.10

[ installation logs of libvirt-snmp including trial and error ]

download libvirt-snmp via git

# git clone git://libvirt.org/libvirt-snmp.git

# ls libvirt-snmp/ AUTHORS  INSTALL.1st  NEWS    autobuild.sh  configure.ac  libvirt-snmp.spec.in COPYING  Makefile.am  README  autogen.sh    docs          src

install some packages which are required to build libvirt-snmp.

# yum install net-snmp-perl net-snmp net-snmp-utils net-snmp-devel libvirt-devel rpm-build -y

build libvirt-snmp

# cd libvirt-snmp/

# ./autobuild.sh test -n "$1" && RESULTS=$1 || RESULTS=results.log : ${AUTOBUILD_INSTALL_ROOT=$HOME/builder} # Make things clean. test -f Makefile && make -k distclean || : rm -rf build mkdir build cd build MIBDIR=$AUTOBUILD_INSTALL_ROOT/share/snmp/mibs mkdir -p $MIBDIR ../autogen.sh --prefix=$AUTOBUILD_INSTALL_ROOT \  --with-mibdir=$MIBDIR ../autogen.sh: line 22: autoreconf: command not found

need autoreconf command to build libvirt-snmp. install autoreconf via yum.

# yum install -y autoconf

run autobuild.sh again.
nn , error

# ./autobuild.sh test -n "$1" && RESULTS=$1 || RESULTS=results.log : ${AUTOBUILD_INSTALL_ROOT=$HOME/builder} # Make things clean. test -f Makefile && make -k distclean || : rm -rf build mkdir build cd build MIBDIR=$AUTOBUILD_INSTALL_ROOT/share/snmp/mibs mkdir -p $MIBDIR ../autogen.sh --prefix=$AUTOBUILD_INSTALL_ROOT \  --with-mibdir=$MIBDIR Can't exec "aclocal": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 326. autoreconf: failed to run aclocal: No such file or directory

I googled this error and this might be solved by installing automake.
install automake

# yum install automake –y

try again. okay.

# ./autobuild.sh <snip> Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-snmp-0.0.3-1.el6.root1358398204.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.Sd7MN0 + umask 022 + cd /root/rpmbuild/BUILD + cd libvirt-snmp-0.0.3 + /bin/rm -rf /root/rpmbuild/BUILDROOT/libvirt-snmp-0.0.3-1.el6.root1358398204.x86_64 + exit 0 Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.SPQKCv + umask 022 + cd /root/rpmbuild/BUILD + rm -rf libvirt-snmp-0.0.3 + exit 0 # echo $? 0 #

libvirt-snmp rpm package will be created under ~/rpmbuild/RPMS/<arch>/ directory.

# ls ~/rpmbuild/RPMS/x86_64/ libvirt-snmp-0.0.3-1.el6.root1358398204.x86_64.rpm

install libvirt-snmp via yum

# yum --nogpgcheck localinstall ~/rpmbuild/RPMS/x86_64/libvirt-snmp-0.0.3-1.el6.root1358398204.x86_64.rpm –y <snip> Running Transaction  Installing : libvirt-snmp-0.0.3-1.el6.root1358398204.x86_64               1/1  Verifying  : libvirt-snmp-0.0.3-1.el6.root1358398204.x86_64               1/1 Installed:  libvirt-snmp.x86_64 0:0.0.3-1.el6.root1358398204 Complete!

check options

# /usr/bin/libvirtMib_subagent --help /usr/bin/libvirtMib_subagent: invalid option -- '-' unknown option ? usage: libvirtGuestTable [-D<tokens>] [-f] [-L] [-M] [-H] [LISTENING ADDRESSES]        -f      Do not fork() from the calling shell.        -DTOKEN[,TOKEN,...]                Turn on debugging output for the given TOKEN(s).                Without any tokens specified, it defaults to printing                all the tokens (which is equivalent to the keyword 'ALL').                You might want to try ALL for extremely verbose output.                Note: You can't put a space between the -D and the TOKENs.        -H      Display a list of configuration file directives                understood by the agent and then exit.        -M      Run as a normal SNMP Agent instead of an AgentX sub-agent.        -x ADDRESS      connect to master agent at ADDRESS (default /var/agentx/master).        -L      Do not open a log file; print all messages to stderr.

display configuration file directives to stderr.

# /usr/bin/libvirtMib_subagent -H -L No log handling enabled - turning on stderr logging Warning: no access control information configured.  It's unlikely this agent can serve any useful purpose in this state.  Run "snmpconf -g basic_setup" to help you configure the libvirtGuestTable.conf file for this agent. Configuration directives understood:  In libvirtGuestTable.conf and libvirtGuestTable.local.conf:    authtrapenable           1 | 2              (1 = enable, 2 = disable)    trapsink                 host [community] [port]    trap2sink                host [community] [port]    informsink               host [community] [port]    trapsess                 [snmpcmdargs] host    trapcommunity            community-string    v1trapaddress            string    agentuser                userid <snip>

a MIB file for libvirt-snmp

# head -5 /usr/share/snmp/mibs/LIBVIRT-MIB.txt LIBVIRT-MIB DEFINITIONS ::= BEGIN -- -- libvirt.org MIB definition --

[ SNMP configuration ]

- snmp get and set configuration

# cd /etc/snmp/ # cp snmpd.conf snmpd.conf.orig

edit snmpd.conf like this:

# egrep -v ^# /etc/snmp/snmpd.conf | grep -v ^$ com2sec notConfigUser  default       public group   notConfigGroup v1           notConfigUser group   notConfigGroup v2c           notConfigUser view    systemview    included   .1 access  notConfigGroup ""      any       noauth    exact  systemview systemview none access MyRWGroup ""      any       noauth    0      all    all    all syslocation Unknown (edit /etc/snmp/snmpd.conf) syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf) dontLogTCPWrappersConnects yes master agentx agentX Socket /var/agentx/master trap2sink localhost public

start snmpd

# /etc/init.d/snmpd restart

run libvirtMib_subagent

# LIBVIRT_DEFAULT_URI="qemu:///system" libvirtMib_subagent -f -L NET-SNMP version 5.5 AgentX subagent connected

try snmpbuldwalk to confirm you can get info about libvrit.

# snmpbulkwalk -v 2c -c public localhost -OX libvirtMIB -m all LIBVIRT-MIB::libvirtGuestName[STRING: bc90f9d2-7366-2c60-891b-23224db5538] = STRING: "centos6-64-1" LIBVIRT-MIB::libvirtGuestState[STRING: bc90f9d2-7366-2c60-891b-23224db5538] = INTEGER: running(1) LIBVIRT-MIB::libvirtGuestCpuCount[STRING: bc90f9d2-7366-2c60-891b-23224db5538] = Gauge32: 1 LIBVIRT-MIB::libvirtGuestMemoryCurrent[STRING: bc90f9d2-7366-2c60-891b-23224db5538] = Gauge32: 1024 LIBVIRT-MIB::libvirtGuestMemoryLimit[STRING: bc90f9d2-7366-2c60-891b-23224db5538] = Gauge32: 1024 LIBVIRT-MIB::libvirtGuestCpuTime[STRING: bc90f9d2-7366-2c60-891b-23224db5538] = Counter64: 119410000000 LIBVIRT-MIB::libvirtGuestRowStatus[STRING: bc90f9d2-7366-2c60-891b-23224db5538] = INTEGER: active(1)

display snmp table

# snmptable -m ALL -v 2c -c public -Cb localhost libvirtGuestTable SNMP table: LIBVIRT-MIB::libvirtGuestTable           Name   State CpuCount MemoryCurrent MemoryLimit      CpuTime RowStatus "centos6-64-1" running        1          1024        1024 120270000000    active

pause the VM “centos6-64-1” via snmpset.

before sending snmpset

# virsh list --all Id    Name                           State ---------------------------------------------------- 1     centos6-64-1                   running

pause the VM via snmp

# snmpset -m ALL -v 2c -c public localhost libvirtGuestState.\'bc90f9d2-7366-2c60-891b-23224db5538\' = paused LIBVIRT-MIB::libvirtGuestState.'....sf,`...2$.U8' = INTEGER: paused(3)

# virsh list --all Id    Name                           State ---------------------------------------------------- 1     centos6-64-1                   paused

run

# snmpset -m ALL -v 2c -c public localhost libvirtGuestState.\'bc90f9d2-7366-2c60-891b-23224db5538\' = running LIBVIRT-MIB::libvirtGuestState.'....sf,`...2$.U8' = INTEGER: running(1) # virsh list --all Id    Name                           State ---------------------------------------------------- 1     centos6-64-1                   running

shutdown

# snmpset -m ALL -v 2c -c public localhost libvirtGuestState.\'bc90f9d2-7366-2c60-891b-23224db5538\' = shutdown

# virsh list --all Id    Name                           State ---------------------------------------------------- -     centos6-64-1                   shut off

start the VM via snmp

at first check UUID

# snmpbulkwalk -v 2c -c public localhost -OX libvirtMIB -m all LIBVIRT-MIB::libvirtGuestName[STRING: bc90f9d2-7366-2c60-891b-23224db5538] = STRING: "centos6-64-1"

start the VM

# virsh list –all Id    Name                           State ---------------------------------------------------- -     centos6-64-1                   shut off # snmpset -m ALL -v 2c -c public localhost libvirtGuestRowStat us.\'bc90f9d2-7366-2c60-891b-23224db5538\' = createAndGo LIBVIRT-MIB::libvirtGuestRowStatus.'....sf,`...2$.U8' = INTEGER: createAndGo(4) # virsh list --all Id    Name                           State ---------------------------------------------------- 2     centos6-64-1                   running

- snmp trap configuration

# cat /etc/snmp/snmptrapd.conf # Example configuration file for snmptrapd # # No traps are handled by default, you must edit this file! # # authCommunity   log,execute,net public # traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold authCommunity log test

run snmptrapd with foreground

# snmptrapd -f -Lo -c /etc/snmp/snmptrapd.conf --disableAuthorization=yes -m ALL NET-SNMP version 5.5 AgentX subagent connected NET-SNMP version 5.5

pause the VM

# snmpset -m ALL -v 2c -c public localhost libvirtGuestState.\'bc90f9d2-7366-2c60-891b-23224db5538\' = paused LIBVIRT-MIB::libvirtGuestState.'....sf,`...2$.U8' = INTEGER: paused(3)

# snmptrapd -f -Lo -c /etc/snmp/snmptrapd.conf --disableAuthorization=yes -m ALL NET-SNMP version 5.5 AgentX subagent connected NET-SNMP version 5.5 2013-01-17 21::01:12 localhost.localdomain [UDP: [127.0.0.1]:43882->[127.0.0.1]]: DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (98514) 0:16:25.14     SNMPv2-MIB::snmpTrapOID.0 = OID: LIBVIRT-MIB::libvirtGuestNotif LIBVIRT-MIB::libvirtGuestName.0 = STRING: "centos6-64-1"        LIBVIRT-MIB::libvirtGuestUUID.1 = STRING: bc90f9d2-7366-2c60-891b-23224db5538   LIBVIRT-MIB::libvirtGuestState.2 = INTEGER: paused(3)   LIBVIRT-MIB::libvirtGuestRowStatus.3 = INTEGER: active(1)

shutdown the VM

# snmpset -m ALL -v 2c -c public localhost libvirtGuestState.\'bc90f9d2-7366-2c60-891b-23224db5538\' = shutdown LIBVIRT-MIB::libvirtGuestState.'....sf,`...2$.U8' = INTEGER: shutdown(4)

trap message

2013-01-17 21::02:41 localhost.localdomain [UDP: [127.0.0.1]:43882->[127.0.0.1]]: DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (107358) 0:17:53.58    SNMPv2-MIB::snmpTrapOID.0 = OID: LIBVIRT-MIB::libvirtGuestNotif LIBVIRT-MIB::libvirtGuestName.0 = STRING: "centos6-64-1"        LIBVIRT-MIB::libvirtGuestUUID.1 = STRING: bc90f9d2-7366-2c60-891b-23224db5538   LIBVIRT-MIB::libvirtGuestState.2 = INTEGER: shutdown(4) LIBVIRT-MIB::libvirtGuestRowStatus.3 = INTEGER: notInService(2) 2013-01-17 21::02:41 localhost.localdomain [UDP: [127.0.0.1]:43882->[127.0.0.1]]: DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (107414) 0:17:54.14    SNMPv2-MIB::snmpTrapOID.0 = OID: LIBVIRT-MIB::libvirtGuestNotif LIBVIRT-MIB::libvirtGuestName.0 = STRING: "centos6-64-1"        LIBVIRT-MIB::libvirtGuestUUID.1 = STRING: bc90f9d2-7366-2c60-891b-23224db5538   LIBVIRT-MIB::libvirtGuestState.2 = INTEGER: shutoff(5)  LIBVIRT-MIB::libvirtGuestRowStatus.3 = INTEGER: notInService(2)

CentOS 6 : how to enable dump core files

# cat /etc/centos-release CentOS release 6.3 (Final) # uname -ri 2.6.32-279.14.1.el6.x86_64 x86_64

edit /etc/sysconfig/init. add the following line.

DAEMON_COREFILE_LIMIT='unlimited'

edit /etc/profile. add the following line

ulimit -c unlimited >/dev/null 2>&1

edit sysctl.conf . specify the directory core dump files are stored.

kernel.core_pattern = '/tmp/core_dump/core'

make the directory for core dump files

# mkdir /tmp/core_dump

reboot the OS

# init 6

how to check whether or not core files are dumped.

This C program will quit immediately and dump a core file.

# cat dump_core.c #include <sys/types.h> #include <unistd.h> #include <signal.h> int main(int argc, char **argv) { kill(getpid(),SIGQUIT); }

compile it.

# gcc dump_core.c -o dump_core.o

execute dump_core.o

# ./dump_core.o Quit (core dumped)

core dump file was created under /tmp directory.

file /tmp/core_dump/core.1708 /tmp/core_dump/core.1708: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from './dump_core.o'

check the dump file with gdb

# gdb ./dump_core.o /tmp/core_dump/core.1708 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.  Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /home/hattori/My_works/dump_core.o...(no debugging symbols found)...done. [New Thread 1708] Missing separate debuginfo for Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/98/5a5bb5fc57c6305be25c76f45d8feff775cc57 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Core was generated by `./dump_core.o'. Program terminated with signal 3, Quit. #0  0x00007f5a2c588b87 in kill () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6_3.6.x86_64 (gdb) quit