2019-05-27 ovs-dpdk

相关链接

• ovs源码：https://github.com/openvswitch/ovs/
• dpdk下载目录：http://fast.dpdk.org/rel/
• ovs+dpdk安装：http://docs.openvswitch.org/en/latest/intro/install/dpdk/

源码安装

• https://github.com/openvswitch/ovs/

编译依赖

---

• yum install make gcc libpcap libpcap-devel kernel-devel kernel-headers kernel.x86_64 -y
• git
• yum install git
• gcc
• yum install gcc
• libssl/openssl(recommended)
• libcap-ng(recommended, Run OVS daemons as a non-root user)
• python2.7
• python six library
• pip install six
• ubound(recommended,DNS)
• autoreconf
• yum install autoconf
• aclocal
• yum install automake
• libtool
• yum install libtool
• 其他
• yum install python-pip

安装依赖

---

• 同编译依赖
• libssl/openssl(recommended)
• libcap-ng(recommended, Run OVS daemons as a non-root user)
• python2.7 and six library
• ubound(recommended,DNS)
• linux kernel>=3.3
• tc
• numactl-devel

编译安装

---

userspace program(ovs-vswitchd/ovsdb)

---

• download source code
• git clone https://github.com/openvswitch/ovs.git
• 切换分支：git branch -d origin/branch-2.9
• 编译
• ./boot.sh
• ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc
• ./configure --with-dpdk=/usr/src/dpdk --prefix=/usr --localstatedir=/var --sysconfdir=/etc
• make
• 安装
• install
• make install
• start(auto script)
• export PATH=$PATH:/usr/share/openvswitch/scripts
• ovs-ctl start
• start(manually)
• ovsdb-tool create /etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
• mkdir -p /var/run/openvswitch
• ovsdb-server --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --log-file
• ovs-vsctl --no-wait init
• ovs-vswitchd --pidfile --detach --log-file
• config
• ovs-vsctl set-controller br0 tcp:127.0.0.1:6633,remote on 6633
• ovs-vsctl set-manager ptcp:6640,listen on 6640
• uninstall
• ovs-ctl stop
• make uninstall
• rm -rf /etc/openvswitch
• rm -rf /usr/share/openvswitch
• rm -rf /var/run/openvswitch/

userspace program with dpdk(ovs-vswitchd/ovsdb)

---

• http://docs.openvswitch.org/en/latest/intro/install/dpdk/

dpdk

• yum install numactl-devel*x86_64
• yum install kernel-devel.x86_64
• cd /usr/src/
• wget http://fast.dpdk.org/rel/dpdk-17.11.4.tar.xz
• tar xf dpdk-17.11.4.tar.xz
• export DPDK_DIR=/usr/src/dpdk-stable-17.11.4
• cd $DPDK_DIR
• export DPDK_TARGET=x86_64-native-linuxapp-gcc
• export DPDK_BUILD=DPDK_TARGET
• make install T=$DPDK_TARGET DESTDIR=install

ovs

• download source code
• git clone https://github.com/openvswitch/ovs.git
• 切换分支：git branch -d origin/branch-2.9
• 编译
• ./boot.sh
• ./configure --with-dpdk=/usr/src/dpdk --prefix=/usr --localstatedir=/var --sysconfdir=/etc
• make
• 安装
• install
• make install
• start(auto script)
• export PATH=$PATH:/usr/share/openvswitch/scripts
• ovs-ctl start
• start(manually)
• ovsdb-tool create /etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
• mkdir -p /var/run/openvswitch
• ovsdb-server --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --log-file
• ovs-vsctl --no-wait init
• huge page
• echo 'vm.nr_hugepages=2048' > /etc/sysctl.d/hugepages.conf
• sysctl -w vm.nr_hugepages=2048
• mount -t hugetlbfs none /dev/hugepages``
• ovs-vswitchd --pidfile --detach --log-file
• config
• ovs-vsctl set-controller br0 tcp:127.0.0.1:6633,remote on 6633
• ovs-vsctl set-manager ptcp:6640,listen on 6640
• ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
• uninstall
• ovs-ctl stop
• make uninstall
• rm -rf /etc/openvswitch
• rm -rf /usr/share/openvswitch
• rm -rf /var/run/openvswitch/

添加网卡

• 查看dpdk网卡绑定情况
  • dpdk-devbind.py --status
• dpdk绑定网卡
  • lspci | grep Eth
  • dpdk-devbind.py --bind=vfio_pci pci_id
  • dpdk-devbind.py --status
• ovs
  • ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
  • ovs-vsctl add-port br0 myportnameone -- set Interface myportnameone type=dpdk options:dpdk-devargs=0000:00:08.0

常用命令

• ovsdb-client dump
• ovs-vsctl
• ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
• ovs-vsctl add-port br0 tap-1 -- set interface tap-1 type=tap ofport_request=1
• ovs-vsctl set port tap-1 tag=100
• ovs-vsctl add-port br0 vhost-user-1 -- set interface vhost-user-1 type=dpdkvhostuser ofport_request=6
• ovs-vsctl add-port phy-br1 phy-br1-br0 -- set interface phy-br1-br0 type=patch options:peer=br0
• ovs-vsctl add-port br0 br0-phy-br1 -- set interface br0-phy-br1 type=patch options:peer=phy-br1
• ovs-vsctl --no-wait set Open_vSwitch . other_config:
• ovs-ofctl
• ovs-appctl
• ovs-dpctl
• ovs-dpctl-top

linux kernel module

---

---

OVS+DPDK

问题

• optimized out：gcc -O0->gcc -O0

ovsdb-server分析

• /etc/openvswitch/conf.db
• json格式
• ovsdb-client dump导出
• 通过/ar/run/openvswitch/db.sock提供服务，ovs-vswitchd通过该socket获取配置信息
• 通过配置的manager提供服务

涉及知识点

• BUILD_BUG_ON
• 定义位置：build_config.h
• 定义内容：
  define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)])
  如果条件为真则引起一个编译时错误。
• FIELD_SIZEOF
• FIELD_SIZEOF获取成员大小
• FIELD_SIZEOF用来获取成员大小。它需要两个参数，第一个指定结构体的类型，第二个则指明成员的名字。
• define FIELD_SIZEOF(t, f) (sizeof(((t*)0)->f))

OVS代码分析

datapath

• datapath/datapath.c
• ovs_nsh_init方法：注册nsh（network service head） offload回调函数到offload_base
• type:ETH_P_NSH
• callbacks:.gso_segment = nsh_gso_segment
• nsh_gso_segment处理流程-----------------------------------------undone
• action_fifos_init方法：分配percpu变量
• 数据结构数组
• 系统的每个cpu对应数组的一个元素。一个cpu不应该访问与其它cpu对应的数组元素，另外，它可以随意读或修改它自己的元素而不用担心出现竞争条件，因为它是唯一有资格这么做的cpu。

ovs-vswitchd

---

进程启动

---

• 入口：ovs-vswitchd.c
• bridge.c:bridge_run()
• init ofproto：初始化ofproto库
  注册ofproto_classes
  init ofproto_classes:注册命令行及其参数和方法
• bridge_run__
  注册dpif_classes
  ofproto_type_run