Add shaker test for cluster in upgrade

Add shaker test plans and test results for ccp cluster during upgrade
for Mitaka to Newton

Change-Id: I5e3fc83048dc28b00a5c10fc50067d2efaf09106

doc/source/test_plans/fuel_ccp_upgrade/plan.rst

@@ -17,13 +17,9 @@ Fuel Containerized Control Plane upgradability performance testing
 Test Plan
 =========
 
-This document is inspired by
+This test plan covers basic network performance with long-running test suites
-:ref:`openstack_control_plane_performance_test_plan`, and aims to cover
+to verify cloud network stability and performance during update Open Stack
-baseline cloud operations and extend this test suite to verify containerized
+from Mitaka to Newton.
-deployment approach. As :ref:`openstack_control_plane_performance_test_plan`
-this test plan covers basic cloud operation VMs creation, as well as
-long-running test suites to verify cloud stability during update Open
-Stack from Mitaka to Newton.
 
 Test Environment
 ----------------

doc/source/test_plans/neutron_fuel_ccp_upgrade/plan.rst

@@ -0,0 +1,189 @@
+.. _neutron_fuel_ccp_upgrade_test_plans:
+
+=============================================================
+OpenStack Neutron Testing During Open Stack Upgrade Test Plan
+=============================================================
+
+:status: **ready**
+:version: 1.0
+
+:Abstract:
+
+  This test plan aims to provide set of tests to identify OpenStack
+  performance against given containerized OpenStack cloud (installed
+  on the top of pre-deployed Kubernetes cluster) using simple minimalistic set
+  of Shaker tests during upgrade from Mitaka to Newton.
+
+:Conventions:
+
+    - **Shaker** - Data plane performance testing tool
+    - **iperf** - Commonly-used network testing tool
+
+
+Test Plan
+=========
+
+The purpose of this section is to describe scenarios for testing Open Stack
+networking during upgrade from Mitaka to Newton.
+
+
+Test Environment
+----------------
+
+Preparation
+^^^^^^^^^^^
+
+This test plan is performed against existing OpenStack cloud installed on top
+of pre-deployed Kubernetes cluster with `fuel-ccp`_ tool with pre-installed Rally
+framework.
+
+.. _fuel-ccp: http://fuel-ccp.readthedocs.io/en/latest/
+
+Environment description
+^^^^^^^^^^^^^^^^^^^^^^^
+The environment description includes hardware specification of servers,
+network parameters, operation system and OpenStack deployment characteristics.
+
+Hardware
+~~~~~~~~
+
+This section contains list of all types of hardware nodes (table below is
+an example).
+
++-----------+-------+----------------------------------------------------+
+| Parameter | Value | Comments                                           |
++-----------+-------+----------------------------------------------------+
+| model     |       | e.g. Supermicro X9SRD-F                            |
++-----------+-------+----------------------------------------------------+
+| CPU       |       | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz |
++-----------+-------+----------------------------------------------------+
+| role      |       | e.g. compute or network                            |
++-----------+-------+----------------------------------------------------+
+
+Network
+~~~~~~~
+
+This section contains list of interfaces and network parameters. For
+complicated cases this section may include topology diagram and switch
+parameters (table below is an example).
+
++------------------+-------+-------------------------+
+| Parameter        | Value | Comments                |
++------------------+-------+-------------------------+
+| network role     |       | e.g. provider or public |
++------------------+-------+-------------------------+
+| card model       |       | e.g. Intel              |
++------------------+-------+-------------------------+
+| driver           |       | e.g. ixgbe              |
++------------------+-------+-------------------------+
+| speed            |       | e.g. 10G or 1G          |
++------------------+-------+-------------------------+
+| MTU              |       | e.g. 9000               |
++------------------+-------+-------------------------+
+| offloading modes |       | e.g. default            |
++------------------+-------+-------------------------+
+
+Software
+~~~~~~~~
+
+This section describes installed software (table below is an example).
+
++-----------------+-------+---------------------------+
+| Parameter       | Value | Comments                  |
++-----------------+-------+---------------------------+
+| OS              |       | e.g. Ubuntu 14.04.3       |
++-----------------+-------+---------------------------+
+| OpenStack       |       | e.g. Mitaka               |
++-----------------+-------+---------------------------+
+| Hypervisor      |       | e.g. KVM                  |
++-----------------+-------+---------------------------+
+| Neutron plugin  |       | e.g. ML2 + OVS            |
++-----------------+-------+---------------------------+
+| L2 segmentation |       | e.g. VLAN / VxLAN / GRE   |
++-----------------+-------+---------------------------+
+| virtual routers |       | e.g. HA / DVR             |
++-----------------+-------+---------------------------+
+
+Test tool
+---------
+
+Shaker wraps around popular system network testing tools like iperf, iperf3
+and netperf (with help of flent). Shaker is able to deploy OpenStack instances
+and networks in different topologies. Shaker scenario specifies the deployment
+and list of tests to execute. Additionally tests may be tuned dynamically
+in command-line.
+
+Test Case 1: Analysis of L2 metrics during Open Stack upgrade
+-------------------------------------------------------------
+
+Description
+^^^^^^^^^^^
+
+`Shaker <http://pyshaker.readthedocs.org/en/latest/index.html>`__ is
+able to deploy OpenStack instances and networks in different topologies.
+
+
+The following steps should be executed:
+
+1. Deploy Kubernetes cluster
+2. Deploy Open Stack cluster
+3. Run full_l2 shaker's scenario and wait successfully stack heats deployment
+4. Start Open Stack upgrade
+
+
+
+List of performance metrics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. table:: Shaker metrics
+
+========  ===============  =================  ======================================
+Priority  Value            Measurement Units  Description
+========  ===============  =================  ======================================
+1         Errors           times              Number of errors during testing
+1         Lost             times              Number of lost results during testing
+1         Latency          ms                 The network latency
+1         TCP bandwidth    Mbits/s            TCP network bandwidth
+2         UDP bandwidth    packets per sec    Number of UDP packets of 32 bytes size
+2         TCP retransmits  packets per sec    Number of retransmitted TCP packets
+========  ===============  =================  ======================================
+
+Test Case 2: Analysis of L3 metrics during Open Stack upgrade
+-------------------------------------------------------------
+
+Description
+^^^^^^^^^^^
+
+`Shaker <http://pyshaker.readthedocs.org/en/latest/index.html>`__ is
+able to deploy OpenStack instances and networks in different topologies.
+
+
+The following steps should be executed:
+
+1. Deploy Kubernetes cluster
+2. Deploy Open Stack cluster
+3. Run full_l3_east_west shaker's scenario and wait successfully stack heats
+   deployment
+4. Start Open Stack upgrade
+
+
+
+List of performance metrics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. table:: Shaker metrics
+
+========  ===============  =================  ======================================
+Priority  Value            Measurement Units  Description
+========  ===============  =================  ======================================
+1         Errors           times              Number of errors during testing
+1         Lost             times              Number of lost results during testing
+1         Latency          ms                 The network latency
+1         TCP bandwidth    Mbits/s            TCP network bandwidth
+2         UDP bandwidth    packets per sec    Number of UDP packets of 32 bytes size
+2         TCP retransmits  packets per sec    Number of retransmitted TCP packets
+========  ===============  =================  ======================================
+
+Reports
+=======
+
+Test plan execution reports:
+ * :ref:`neutron_fuel_ccp_upgrade_report`

doc/source/test_results/neutron_fuel_ccp_upgrade/configs/ccp.yaml

@@ -0,0 +1,17 @@
+builder:
+  push: true
+  no_cache: false
+registry:
+  address: "172.20.9.32:5000/env-1"
+repositories:
+  skip_empty: True
+kubernetes:
+  server: http://127.0.0.1:8080
+  image_pull_policy: Always
+---
+!include
+- versions.yaml
+- topology.yaml
+- configs.yaml
+- repos.yaml
+

doc/source/test_results/neutron_fuel_ccp_upgrade/configs/configs.yaml

@@ -0,0 +1,64 @@
+configs:
+  private_interface: p1p1.602
+  public_interface: p1p1.602
+  ingress:
+    enabled: true
+  glance:
+    tls:
+      enabled: false
+    bootstrap:
+      enable: true
+  neutron:
+    tls:
+      enabled: false
+    physnets:
+      - name: "physnet1"
+        bridge_name: "br-external"
+        interface: "bond0.648"
+        flat: true
+        vlan_range: false
+        dpdk: false
+    bootstrap:
+      internal:
+        enable: true
+      external:
+        enable: true
+        net_name: ext-net
+        subnet_name: ext-subnet
+        physnet: physnet1
+        network: 172.20.172.0/22
+        gateway: 172.20.172.1
+        nameserver: 8.8.8.8
+        pool:
+          start: 172.20.173.0
+          end: 172.20.175.254
+  keystone:
+    debug: true
+  security:
+    tls:
+      create_certificates: false
+  etcd:
+    tls:
+      enabled: false
+  db:
+    tls:
+      enabled: false
+  rabbitmq:
+    tls:
+      enabled: false
+  keystone:
+    tls:
+      enabled: false
+  cinder:
+    tls:
+      enabled: false
+  heat:
+    tls:
+      enabled: false
+  nova:
+    tls:
+      enabled: false
+  horizon:
+    tls:
+      enabled: false
+

doc/source/test_results/neutron_fuel_ccp_upgrade/configs/repos.yaml

@@ -0,0 +1,39 @@
+repositories:
+  repos:
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-ceph
+    name: fuel-ccp-ceph
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-debian-base
+    name: fuel-ccp-debian-base
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-entrypoint
+    name: fuel-ccp-entrypoint
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-etcd
+    name: fuel-ccp-etcd
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-galera
+    name: fuel-ccp-galera
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-glance
+    name: fuel-ccp-glance
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-heat
+    name: fuel-ccp-heat
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-horizon
+    name: fuel-ccp-horizon
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-ironic
+    name: fuel-ccp-ironic
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-keystone
+    name: fuel-ccp-keystone
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-mariadb
+    name: fuel-ccp-mariadb
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-memcached
+    name: fuel-ccp-memcached
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-murano
+    name: fuel-ccp-murano
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-neutron
+    name: fuel-ccp-neutron
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-nginx
+    name: fuel-ccp-nginx
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-nova
+    name: fuel-ccp-nova
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-openstack-base
+    name: fuel-ccp-openstack-base
+  - git_url: https://git.openstack.org/openstack/fuel-ccp-rabbitmq
+    name: fuel-ccp-rabbitmq
+

doc/source/test_results/neutron_fuel_ccp_upgrade/configs/topology.yaml

@@ -0,0 +1,86 @@
+nodes:
+  # node[1-3]: ETCD
+  # node[4-5]: Kubernetes master
+
+  node([6|7|8])$: # 6-8
+    roles:
+      - controller
+      - openvswitch
+      - messaging
+      - db
+      - indentity
+
+  node9$:
+    roles:
+      - etc
+      - backups
+      - indentity
+
+  node[1-9][0-9]$: # 10-99
+    roles:
+      - compute
+      - openvswitch
+      - indentity
+
+replicas:
+  database: 3
+  etcd: 1
+  rpc: 3
+  notifications: 3
+  glance-api: 1
+  glance-registry: 1
+  keystone: 3
+  nova-api: 3
+  nova-scheduler: 3
+  nova-conductor: 3
+  neutron-server: 3
+  neutron-metadata-agent: 3
+  horizon: 3
+  heat-api: 3
+  heat-engine: 3
+
+roles:
+  db:
+    - database
+  messaging:
+    - rpc
+    - notifications
+  controller:
+    - etcd
+    - glance-api
+    - glance-registry
+    - heat-api-cfn
+    - heat-api
+    - heat-engine
+    - horizon
+    - memcached
+    - neutron-dhcp-agent
+    - neutron-l3-agent
+    - neutron-metadata-agent
+    - neutron-server
+    - nova-api
+    - nova-conductor
+    - nova-consoleauth
+    - nova-novncproxy
+    - nova-scheduler
+  compute:
+    - nova-compute
+    - nova-libvirt
+  openvswitch:
+    - neutron-openvswitch-agent
+    - openvswitch-db
+    - openvswitch-vswitchd
+  etc:
+    - etcd
+  indentity:
+    - keystone
+  backups:
+    - backup
+services:
+  database:
+    service_def: galera
+  rpc:
+    service_def: rabbitmq
+  notifications:
+    service_def: rabbitmq
+

doc/source/test_results/neutron_fuel_ccp_upgrade/configs/versions-mitaka.yaml

@@ -0,0 +1,41 @@
+images:
+  tag: mitaka
+sources:
+  openstack/cinder:
+    git_ref: stable/mitaka
+    git_url: https://github.com/openstack/cinder.git
+  openstack/glance:
+    git_ref: stable/mitaka
+    git_url: https://github.com/openstack/glance.git
+  openstack/heat:
+    git_ref: stable/mitaka
+    git_url: https://github.com/openstack/heat.git
+  openstack/horizon:
+    git_ref: stable/mitaka
+    git_url: https://github.com/openstack/horizon.git
+  openstack/keystone:
+    git_ref: stable/mitaka
+    git_url: https://github.com/openstack/keystone.git
+  openstack/neutron:
+    git_ref: stable/mitaka
+    git_url: https://github.com/openstack/neutron.git
+  openstack/nova:
+    git_ref: stable/mitaka
+    git_url: https://github.com/openstack/nova.git
+  openstack/requirements:
+    git_ref: stable/mitaka
+    git_url: https://git.openstack.org/openstack/requirements.git
+  openstack/sahara-dashboard:
+    git_ref: stable/mitaka
+    git_url: https://git.openstack.org/openstack/sahara-dashboard.git
+  openstack/murano-dashboard:
+    git_url: http://git.openstack.org/openstack/murano-dashboard.git
+    git_ref: stable/mitaka
+  openstack/searchlight-ui:
+    git_url: https://git.openstack.org/openstack/searchlight-ui.git
+    git_ref: stable/mitaka
+  openstack/designate-dashboard:
+    git_url: https://git.openstack.org/openstack/designate-dashboard.git
+    git_ref: stable/mitaka
+
+

doc/source/test_results/neutron_fuel_ccp_upgrade/configs/versions-newton.yaml

@@ -0,0 +1,81 @@
+images:
+  tag: mitaka
+  image_specs:
+    keystone:
+      tag: newton
+
+    horizon:
+      tag: newton
+
+    nova-upgrade:
+      tag: newton
+    nova-api:
+      tag: newton
+    nova-conductor:
+      tag: newton
+    nova-consoleauth:
+      tag: newton
+    nova-novncproxy:
+      tag: newton
+    nova-scheduler:
+      tag: newton
+    nova-compute:
+      tag: newton
+    nova-libvirt:
+      tag: newton
+
+    neutron-dhcp-agent:
+      tag: newton
+    neutron-l3-agent:
+      tag: newton
+    neutron-metadata-agent:
+      tag: newton
+    neutron-server:
+      tag: newton
+    neutron-openvswitch-agent:
+      tag: newton
+
+    glance-api:
+      tag: newton
+    glance-registry:
+      tag: newton
+    glance-upgrade:
+      tag: newton
+sources:
+  openstack/cinder:
+    git_ref: stable/newton
+    git_url: https://github.com/openstack/cinder.git
+  openstack/glance:
+    git_ref: stable/newton
+    git_url: https://github.com/openstack/glance.git
+  openstack/heat:
+    git_ref: stable/newton
+    git_url: https://github.com/openstack/heat.git
+  openstack/horizon:
+    git_ref: stable/newton
+    git_url: https://github.com/openstack/horizon.git
+  openstack/keystone:
+    git_ref: stable/newton
+    git_url: https://github.com/openstack/keystone.git
+  openstack/neutron:
+    git_ref: stable/newton
+    git_url: https://github.com/openstack/neutron.git
+  openstack/nova:
+    git_ref: stable/newton
+    git_url: https://github.com/openstack/nova.git
+  openstack/requirements:
+    git_ref: stable/newton
+    git_url: https://git.openstack.org/openstack/requirements.git
+  openstack/sahara-dashboard:
+    git_ref: stable/newton
+    git_url: https://git.openstack.org/openstack/sahara-dashboard.git
+  openstack/murano-dashboard:
+    git_url: http://git.openstack.org/openstack/murano-dashboard.git
+    git_ref: stable/newton
+  openstack/searchlight-ui:
+    git_url: https://git.openstack.org/openstack/searchlight-ui.git
+    git_ref: stable/newton
+  openstack/designate-dashboard:
+    git_url: https://git.openstack.org/openstack/designate-dashboard.git
+    git_ref: stable/newton
+

doc/source/test_results/neutron_fuel_ccp_upgrade/index.rst

@@ -0,0 +1,141 @@
+.. _neutron_fuel_ccp_upgrade_report:
+
+==================================================================
+Fuel Containerized Control Plane performance report during upgrade
+==================================================================
+
+:Abstract:
+
+  This document includes OpenStack Networking (aka Neutron) performance test
+  results during upgrade from Mitaka to Newton. All tests have been performed
+  regarding
+  :ref:`neutron_fuel_ccp_upgrade_test_plans`
+
+Environment description
+=======================
+
+For Kubernetes pre-deployment `Kargo tool`_ was used. More information about
+*fuel-ccp* and how it can be installed can be found in
+`official documentation`_.
+
+.. _Kargo tool: https://github.com/kubespray/kargo
+.. _official documentation: http://fuel-ccp.readthedocs.io/en/latest/
+
+Hardware configuration of each server
+-------------------------------------
+
+All servers have configuration describing in table below
+
+.. table:: Description of servers hardware
+
+   +-------+----------------+-------------------------------+
+   |server |vendor,model    |HP,DL380 Gen9                  |
+   +-------+----------------+-------------------------------+
+   |CPU    |vendor,model    |Intel,E5-2680 v3               |
+   |       +----------------+-------------------------------+
+   |       |processor_count |2                              |
+   |       +----------------+-------------------------------+
+   |       |core_count      |12                             |
+   |       +----------------+-------------------------------+
+   |       |frequency_MHz   |2500                           |
+   +-------+----------------+-------------------------------+
+   |RAM    |vendor,model    |HP,752369-081                  |
+   |       +----------------+-------------------------------+
+   |       |amount_MB       |262144                         |
+   +-------+----------------+-------------------------------+
+   |NETWORK|interface_name  |ens4f1                         |
+   |       +----------------+-------------------------------+
+   |       |vendor,model    |Intel,X710 Dual Port           |
+   |       +----------------+-------------------------------+
+   |       |bandwidth       |10G                            |
+   |       +----------------+-------------------------------+
+   |       |interface_name  |p1p1                           |
+   |       +----------------+-------------------------------+
+   |       |vendor,model    |Intel,X710 Dual Port           |
+   |       +----------------+-------------------------------+
+   |       |bandwidth       |10G                            |
+   +-------+----------------+-------------------------------+
+   |STORAGE|dev_name        |/dev/sda                       |
+   |       +----------------+-------------------------------+
+   |       |vendor,model    | | raid10 - HP P840            |
+   |       |                | | 12 disks EH0600JEDHE        |
+   |       +----------------+-------------------------------+
+   |       |SSD/HDD         |HDD                            |
+   |       +----------------+-------------------------------+
+   |       |size            | 3,6TB                         |
+   +-------+----------------+-------------------------------+
+
+
+Network configuration of each server
+------------------------------------
+
+All servers have same network configuration:
+
+.. image:: configs/Network_Scheme.png
+   :alt: Network Scheme of the environment
+   :width: 650px
+
+Fuel CCP configuration
+----------------------
+CCP main configuration
+:download:`ccp.yaml <configs/ccp.yaml>`
+
+CCP Cluster configuration
+:download:`configs.yaml <configs/configs.yaml>`
+
+CCP Cluster topology
+:download:`topology.yaml <configs/topology.yaml>`
+
+CCP repositories configuration
+:download:`repos.yaml <configs/repos.yaml>`
+
+CCP versions configuration for Mitaka
+:download:`versions-mitaka.yaml <configs/versions-mitaka.yaml>`
+
+CCP versions configuration for upgrade from Mitaka to Newton
+:download:`versions-newton.yaml <configs/versions-newton.yaml>`
+
+Upgrade and tests procedure
+---------------------------
+* Install fuel-ccp
+* Rename ``ccp.yaml`` to ``/root/.ccp.ymal``
+* Copy all yaml files to ``/root/``
+* Rename ``versions-mitaka.yaml`` to ``versions.yaml``
+* Execute ``ccp deploy`` and wait ok status for all services
+* Install shaker
+* On the node 172.20.9.40 execute ``shaker --scenario openstack/full_l2 --os-insecure --report full-l2.html --server-endpoint 172.20.9.40:31998``
+* Wait message ``INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.JoinOperation...`` from shaker
+* Rename ``versions-newton.yaml`` to ``versions.yaml``
+* Execute ``ccp deploy`` and wait ok status for all services
+* Wait tests results from shaker
+
+Test results
+============
+Shaker provides statistics about maximum, minimum and mean values of different
+connection measurements. For each test was found the maximum among all maximum
+values, minimum among all minimum values and counts the mean value from all
+mean values.
+
+
+Test Case 1: Analysis of L2 metrics during Open Stack upgrade
+-------------------------------------------------------------
+.. table:: Full L2 test results
+
+   +-------+----------------+---------------------------------------------------------+
+   |Losts  |Errors          |Full report                                              |
+   +-------+----------------+---------------------------------------------------------+
+   |10     |0               |:download:`L2 Full tests report <reports/full-l2.html>`  |
+   +-------+----------------+---------------------------------------------------------+
+
+
+
+Test Case 2: Analysis of L3 metrics during Open Stack upgrade
+-------------------------------------------------------------
+.. table:: Full L3 East West test results
+
+   +-------+----------------+-----------------------------------------------------------------------------+
+   |Losts  |Errors          |Full report                                                                  |
+   +-------+----------------+-----------------------------------------------------------------------------+
+   |0      |11              |:download:`L3 East West Full tests report <reports/full-l3-east-west.html>`  |
+   +-------+----------------+-----------------------------------------------------------------------------+
+