OpenStack logs ( network namespace )

Reference

https://www.rdoproject.org/troubleshooting/networking/

https://www.rdoproject.org/networking/networking-in-too-much-detail/

There are three nodes running, controller, network and compute node in my environment.

                  the internet

vm1  vm2            |

compute          network node            controller node

   |------------------------|-------------------------------------|

user01 has a router “my-router”.

+--------------------------------------+-----------+----------------------------------------------------------------------+ (neutron) [root@controller ~(keystone_user01)]# neutron router-list +--------------------------------------+-----------+----------------------------------------------------------------------+ | id                                   | name      | external_gateway_info                                                | +--------------------------------------+-----------+----------------------------------------------------------------------+ | 3be9346b-9447-46d4-8222-f507be59357e | my-router | {"network_id": "b586f891-9821-4b0b-8d0e-b224b164383c",               | |                                      |           | "enable_snat": true, "external_fixed_ips": [{"subnet_id": "96a2cf60  | |                                      |           | -346d-4ad7-8975-89a4c8f19bb0", "ip_address": "192.168.131.131"}]}    | +--------------------------------------+-----------+----------------------------------------------------------------------+ [root@controller ~(keystone_user01)]#

on the network node, list network namespace.

[root@network ~]# ip netns list qdhcp-b586f891-9821-4b0b-8d0e-b224b164383c qdhcp-7745f545-4d41-4f0d-9653-899a1526af8c qrouter-3be9346b-9447-46d4-8222-f507be59357e qdhcp-ecc3c3f9-31bd-4998-a41d-b682b508bfb5 qdhcp-d586146c-d5cb-42d5-b2f6-c0a78a7e8f82 qrouter-cf414122-2ed7-446c-88a6-448876d8196b

qrouter- : router’s namespace

qdhcp- : dhcp namespace

[ router namespace ]

my-router has two interfaces.

qr-ec8a8119-49 is for internal network

qg-05a86ecc-dd is for external network

[root@network ~]# ip netns exec qrouter-3be9346b-9447-46d4-8222-f507be59357e ip -4 a s 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever 12: qr-ec8a8119-49: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN    inet 192.168.37.1/24 brd 192.168.37.255 scope global qr-ec8a8119-49       valid_lft forever preferred_lft forever 19: qg-05a86ecc-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN    inet 192.168.131.131/24 brd 192.168.131.255 scope global qg-05a86ecc-dd       valid_lft forever preferred_lft forever    inet 192.168.131.132/32 brd 192.168.131.132 scope global qg-05a86ecc-dd       valid_lft forever preferred_lft forever    inet 192.168.131.133/32 brd 192.168.131.133 scope global qg-05a86ecc-dd       valid_lft forever preferred_lft forever

Access to the instance from this router.

Two instances are running.

[root@controller ~(keystone_user01)]# nova list +--------------------------------------+-------------------+--------+------------+-------------+----------------------------------------+ | ID                                   | Name              | Status | Task State | Power State | Networks                               | +--------------------------------------+-------------------+--------+------------+-------------+----------------------------------------+ | 76dbd219-4e69-461e-9404-26270e9b4cd8 | my-first-instance | ACTIVE | -          | Running     | internal=192.168.37.3, 192.168.131.132 | | c98521fc-cf0a-4dc1-ac89-4f88bb5d80c8 | second-instance   | ACTIVE | -          | Running     | internal=192.168.37.4, 192.168.131.133 | +--------------------------------------+-------------------+--------+------------+-------------+----------------------------------------+ [root@controller ~(keystone_user01)]#

Access to the instance over SSH from qrouter.

[root@network ~]# ip netns exec qrouter-3be9346b-9447-46d4-8222-f507be59357e ssh cirros@192.168.37.3 cirros@192.168.37.3's password: $ hostname my-first-instance $ exit Connection to 192.168.37.3 closed. [root@network ~]#

[ DHCP namespace ]

[root@controller ~(keystone_user01)]# neutron net-list +--------------------------------------+----------+-------------------------------------------------------+ | id                                   | name     | subnets                                               | +--------------------------------------+----------+-------------------------------------------------------+ | b586f891-9821-4b0b-8d0e-b224b164383c | ext      | 96a2cf60-346d-4ad7-8975-89a4c8f19bb0 192.168.131.0/24 | | f6d28e0c-0233-416e-a212-b1d36d4d3cbf | public   | 22f91b90-5c43-474e-8e01-b72ac359186e                  | | ecc3c3f9-31bd-4998-a41d-b682b508bfb5 | internal | 37b936a4-cd4c-4565-9dee-2961cfc592c7 192.168.37.0/24  | +--------------------------------------+----------+-------------------------------------------------------+ [root@controller ~(keystone_user01)]#

DHCP namespace for internal network.

[root@network ~]# ip netns exec qdhcp-ecc3c3f9-31bd-4998-a41d-b682b508bfb5 ip -4  a s 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever 11: tap58c34b1f-66: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN    inet 192.168.37.2/24 brd 192.168.37.255 scope global tap58c34b1f-66       valid_lft forever preferred_lft forever [root@network ~]# [root@network ~]# ip netns exec qdhcp-ecc3c3f9-31bd-4998-a41d-b682b508bfb5 ip -4 r s default via 192.168.37.1 dev tap58c34b1f-66 192.168.37.0/24 dev tap58c34b1f-66  proto kernel  scope link  src 192.168.37.2 [root@network ~]#

DHCP namespace for external network.

[root@network ~]# ip netns exec qdhcp-b586f891-9821-4b0b-8d0e-b224b164383c ip -4 a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever 18: tap89b14979-b3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN    inet 192.168.131.130/24 brd 192.168.131.255 scope global tap89b14979-b3       valid_lft forever preferred_lft forever [root@network ~]# ip netns exec qdhcp-b586f891-9821-4b0b-8d0e-b224b164383c ip -4 r s default via 192.168.131.1 dev tap89b14979-b3 192.168.131.0/24 dev tap89b14979-b3  proto kernel  scope link  src 192.168.131.130 [root@network ~]#

Here is a dnsmasq process for external network.

[root@network ~]# ps aux | grep dnsmasq | grep ap89b14979-b3 | grep -v grep nobody   22575  0.0  0.0  15556   872 ?        S     9月14   0:00 dnsmasq --no-hosts --no-resolv --strict-order --except-interface=lo --pid-file=/var/lib/neutron/dhcp/b586f891-9821-4b0b-8d0e-b224b164383c/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/b586f891-9821-4b0b-8d0e-b224b164383c/host --addn-hosts=/var/lib/neutron/dhcp/b586f891-9821-4b0b-8d0e-b224b164383c/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/b586f891-9821-4b0b-8d0e-b224b164383c/opts --dhcp-leasefile=/var/lib/neutron/dhcp/b586f891-9821-4b0b-8d0e-b224b164383c/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=tap89b14979-b3 --dhcp-range=set:tag0,192.168.131.0,static,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq-neutron.conf --domain=openstacklocal [root@network ~]#

this is for internal network.

[root@network ~]# ps aux | grep dnsmasq | grep tap58c34b1f-66 | grep -v grep nobody   22585  0.0  0.0  15556   956 ?        S     9月14   0:00 dnsmasq --no-hosts --no-resolv --strict-order --except-interface=lo --pid-file=/var/lib/neutron/dhcp/ecc3c3f9-31bd-4998-a41d-b682b508bfb5/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/ecc3c3f9-31bd-4998-a41d-b682b508bfb5/host --addn-hosts=/var/lib/neutron/dhcp/ecc3c3f9-31bd-4998-a41d-b682b508bfb5/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/ecc3c3f9-31bd-4998-a41d-b682b508bfb5/opts --dhcp-leasefile=/var/lib/neutron/dhcp/ecc3c3f9-31bd-4998-a41d-b682b508bfb5/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=tap58c34b1f-66 --dhcp-range=set:tag0,192.168.37.0,static,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq-neutron.conf --domain=openstacklocal [root@network ~]#

LXC, dnsmasq : How to configure dnsmasq to have dnsmasq add MAC address of containers when dnsmasq sends DNS queries to other DNS servers.

hattori@ubuntu:~$ tail -1 /etc/lsb-release

DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"

hattori@ubuntu:~$

hattori@ubuntu:~$ dnsmasq --version

Dnsmasq version 2.68  Copyright (c) 2000-2013 Simon Kelley

By adding MAC address in DNS queries, DNS servers could identify devices even though those devices are behind NAT environment by looking at MAC address in DNS queries. ( If DNS servers can recognize that. )

dnsmasq can add MAC address in DNS queries in EDNS0.

man dnsmasq

      --add-mac

             Add the MAC address of the requestor to DNS  queries  which  are

             forwarded  upstream.  This  may  be used to DNS filtering by the

             upstream server. The MAC  address  can  only  be  added  if  the

             requestor is on the same subnet as the dnsmasq server. Note that

             the mechanism used to achieve this (an EDNS0 option) is not  yet

             standardised,  so  this  should be considered experimental. Also

             note that exposing MAC addresses in this way may  have  security

             and  privacy  implications.  The warning about caching given for

             --add-subnet applies to --add-mac too.

By default, dnsmasq does not add MAC address in DNS queries.

dnsmasq configuration file for LXC(lxc-net) is /etc/init/lxc-net.conf.

So I modified that so that dnsmasq can add MAC address into DNS queries.

Copy a current config file.

hattori@ubuntu:~$ sudo cp /etc/init/lxc-net.conf ./lxc-net.conf.20151024

Edit lxc-net.conf.

Add  '--add-mac' option in that file.

$ diff /etc/init/lxc-net.conf lxc-net.conf.20151024

65c65

< dnsmasq $LXC_DOMAIN_ARG -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=${varrun}/dnsmasq.pid --conf-file=${LXC_DHCP_CONFILE} --listen-address ${LXC_ADDR} --dhcp-range ${LXC_DHCP_RANGE} --dhcp-lease-max=${LXC_DHCP_MAX} --dhcp-no-override --except-interface=lo --interface=${LXC_BRIDGE} --dhcp-leasefile=/var/lib/misc/dnsmasq.${LXC_BRIDGE}.leases --dhcp-authoritative --add-mac || cleanup

---

> dnsmasq $LXC_DOMAIN_ARG -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=${varrun}/dnsmasq.pid --conf-file=${LXC_DHCP_CONFILE} --listen-address ${LXC_ADDR} --dhcp-range ${LXC_DHCP_RANGE} --dhcp-lease-max=${LXC_DHCP_MAX} --dhcp-no-override --except-interface=lo --interface=${LXC_BRIDGE} --dhcp-leasefile=/var/lib/misc/dnsmasq.${LXC_BRIDGE}.leases --dhcp-authoritative || cleanup

stop and start lxc-net to reflect that change.

check current PID of dnsmasq.

hattori@ubuntu:~$ cat /var/run/lxc/dnsmasq.pid

1480

Here are current options.

hattori@ubuntu:~$ ps aux | grep 1480 | grep -v grep

lxc-dns+  1480  0.0  0.0  28204   968 ?        S    10月23   0:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative

root      1533  0.0  0.0 450312 14808 ?        Sl   10月23   0:00 /usr/sbin/libvirtd -d

stop and start lxc-net.

hattori@ubuntu:~$ sudo stop lxc-net

lxc-net stop/waiting

hattori@ubuntu:~$ sudo start lxc-net

lxc-net start/running

hattori@ubuntu:~$ cat /var/run/lxc/dnsmasq.pid

6558

dnsmasq has started with --add-mac option.

hattori@ubuntu:~$ ps aux | grep 6558 | grep -v grep

lxc-dns+  6558  0.0  0.0  28204   960 ?        S    00:59   0:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative --add-mac

Start a container.

hattori@ubuntu:~$ sudo lxc-start -n ubuntu01 -d

Connect to the container and check MAC address of that container.

hattori@ubuntu:~$ sudo lxc-ls -f

NAME             STATE    IPV4       IPV6  AUTOSTART  

----------------------------------------------------

ubuntu-original  STOPPED  -          -     NO         

ubuntu01         RUNNING  10.0.3.47  -     NO         

ubuntu02         STOPPED  -          -     NO         

ubuntu03         STOPPED  -          -     NO         

ubuntu04         STOPPED  -          -     NO         

hattori@ubuntu:~$

hattori@ubuntu:~$ ssh 10.0.3.47 -l ubuntu

ubuntu@10.0.3.47's password:

Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-66-generic x86_64)

* Documentation:  https://help.ubuntu.com/

Last login: Sat Oct 24 00:26:17 2015 from 10.0.3.1

ubuntu@ubuntu01:~$

Here is a resolv.conf of the container.

This container sends DNS queries to dnsmasq(10.0.3.1).

root@ubuntu01:~# cat /etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)

#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 10.0.3.1

root@ubuntu01:~#

Here is a MAC address of the container.

root@ubuntu01:~# ip addr show eth0 | grep 'link/ether'

   link/ether 00:16:3e:09:dc:c2 brd ff:ff:ff:ff:ff:ff

Here is a capture data collected on LXC host when dnsmasq sent DNS queries to other DNS servers.

Having looked at the capture data, you can see MAC address in DNS queries which was added by dnsmasq.

    Queries

       www.google.com: type A, class IN

           Name: www.google.com

           Type: A (Host address)

           Class: IN (0x0001)

   Additional records

       <Root>: type OPT

           Name: <Root>

           Type: OPT (EDNS0 option)

           UDP payload size: 4096

           Higher bits in extended RCODE: 0x0

           EDNS0 version: 0

           Z: 0x0

           Data length: 10

           Option: Unknown (65001)

               Option Code: Unknown (65001)

               Option Length: 6

               Option Data: 00163e09dcc2