diff --git a/doc/source/admin/configuration/hypervisor-kvm.rst b/doc/source/admin/configuration/hypervisor-kvm.rst index 3e2f94a128a8..569e4e17be5e 100644 --- a/doc/source/admin/configuration/hypervisor-kvm.rst +++ b/doc/source/admin/configuration/hypervisor-kvm.rst @@ -289,154 +289,6 @@ On the Nova compute node in nova.conf: rbd_ceph_conf=/etc/ceph/ceph.conf rbd_connect_timeout=5 -Specify the CPU model of KVM guests -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The Compute service enables you to control the guest CPU model that is exposed -to KVM virtual machines. Use cases include: - -* To maximize performance of virtual machines by exposing new host CPU features - to the guest - -* To ensure a consistent default CPU across all machines, removing reliance of - variable QEMU defaults - -In libvirt, the CPU is specified by providing a base CPU model name (which is a -shorthand for a set of feature flags), a set of additional feature flags, and -the topology (sockets/cores/threads). The libvirt KVM driver provides a number -of standard CPU model names. These models are defined in the -``/usr/share/libvirt/cpu_map.xml`` file for libvirt prior to version 4.7.0 or -``/usr/share/libvirt/cpu_map/*.xml`` files thereafter. Make a check to -determine which models are supported by your local installation. - -Two Compute configuration options in the :oslo.config:group:`libvirt` group -of ``nova.conf`` define which type of CPU model is exposed to the hypervisor -when using KVM: :oslo.config:option:`libvirt.cpu_mode` and -:oslo.config:option:`libvirt.cpu_models`. - -The :oslo.config:option:`libvirt.cpu_mode` option can take one of the following -values: ``none``, ``host-passthrough``, ``host-model``, and ``custom``. - -See `Effective Virtual CPU configuration in Nova`_ for a recorded presentation -about this topic. - -.. _Effective Virtual CPU configuration in Nova: https://www.openstack.org/videos/summits/berlin-2018/effective-virtual-cpu-configuration-in-nova - -Host model (default for KVM & QEMU) ------------------------------------ - -If your ``nova.conf`` file contains ``cpu_mode=host-model``, libvirt identifies -the CPU model in ``/usr/share/libvirt/cpu_map.xml`` for version prior to 4.7.0 -or ``/usr/share/libvirt/cpu_map/*.xml`` for version 4.7.0 and higher that most -closely matches the host, and requests additional CPU flags to complete the -match. This configuration provides the maximum functionality and performance -and maintains good reliability. - -With regard to enabling and facilitating live migration between -compute nodes, you should assess whether ``host-model`` is suitable -for your compute architecture. In general, using ``host-model`` is a -safe choice if your compute node CPUs are largely identical. However, -if your compute nodes span multiple processor generations, you may be -better advised to select a ``custom`` CPU model. - -Host pass through ------------------ - -If your ``nova.conf`` file contains ``cpu_mode=host-passthrough``, libvirt -tells KVM to pass through the host CPU with no modifications. The difference -to host-model, instead of just matching feature flags, every last detail of the -host CPU is matched. This gives the best performance, and can be important to -some apps which check low level CPU details, but it comes at a cost with -respect to migration. - -In ``host-passthrough`` mode, the guest can only be live-migrated to a -target host that matches the source host extremely closely. This -definitely includes the physical CPU model and running microcode, and -may even include the running kernel. Use this mode only if - -* your compute nodes have a very large degree of homogeneity - (i.e. substantially all of your compute nodes use the exact same CPU - generation and model), and you make sure to only live-migrate - between hosts with exactly matching kernel versions, *or* - -* you decide, for some reason and against established best practices, - that your compute infrastructure should not support any live - migration at all. - -Custom ------- - -If :file:`nova.conf` contains :oslo.config:option:`libvirt.cpu_mode`\ =custom, -you can explicitly specify an ordered list of supported named models using -the :oslo.config:option:`libvirt.cpu_models` configuration option. It is -expected that the list is ordered so that the more common and less advanced cpu -models are listed earlier. - -An end user can specify required CPU features through traits. When specified, -the libvirt driver will select the first cpu model in the -:oslo.config:option:`libvirt.cpu_models` list that can provide the requested -feature traits. If no CPU feature traits are specified then the instance will -be configured with the first cpu model in the list. - -For example, if specifying CPU features ``avx`` and ``avx2`` as follows: - -.. code-block:: console - - $ openstack flavor set FLAVOR_ID --property trait:HW_CPU_X86_AVX=required \ - --property trait:HW_CPU_X86_AVX2=required - -and :oslo.config:option:`libvirt.cpu_models` is configured like this: - -.. code-block:: ini - - [libvirt] - cpu_mode = custom - cpu_models = Penryn,IvyBridge,Haswell,Broadwell,Skylake-Client - -Then ``Haswell``, the first cpu model supporting both ``avx`` and ``avx2``, -will be chosen by libvirt. - -In selecting the ``custom`` mode, along with a -:oslo.config:option:`libvirt.cpu_models` that matches the oldest of your compute -node CPUs, you can ensure that live migration between compute nodes will always -be possible. However, you should ensure that the -:oslo.config:option:`libvirt.cpu_models` you select passes the correct CPU -feature flags to the guest. - -If you need to further tweak your CPU feature flags in the ``custom`` -mode, see `Set CPU feature flags`_. - -.. note:: - - If :oslo.config:option:`libvirt.cpu_models` is configured, - the CPU models in the list needs to be compatible with the host CPU. Also, if - :oslo.config:option:`libvirt.cpu_model_extra_flags` is configured, all flags - needs to be compatible with the host CPU. If incompatible CPU models or flags - are specified, nova service will raise an error and fail to start. - - -None (default for all libvirt-driven hypervisors other than KVM & QEMU) ------------------------------------------------------------------------ - -If your ``nova.conf`` file contains ``cpu_mode=none``, libvirt does not specify -a CPU model. Instead, the hypervisor chooses the default model. - -Set CPU feature flags -~~~~~~~~~~~~~~~~~~~~~ - -Regardless of whether your selected :oslo.config:option:`libvirt.cpu_mode` is -``host-passthrough``, ``host-model``, or ``custom``, it is also -possible to selectively enable additional feature flags. Suppose your -selected ``custom`` CPU model is ``IvyBridge``, which normally does -not enable the ``pcid`` feature flag --- but you do want to pass -``pcid`` into your guest instances. In that case, you would set: - -.. code-block:: ini - - [libvirt] - cpu_mode = custom - cpu_models = IvyBridge - cpu_model_extra_flags = pcid Nested guest support ~~~~~~~~~~~~~~~~~~~~ @@ -446,8 +298,8 @@ your Nova instances to themselves run hardware-accelerated virtual machines with KVM. Doing so requires a module parameter on your KVM kernel module, and corresponding ``nova.conf`` settings. -Nested guest support in the KVM kernel module ---------------------------------------------- +Host configuration +------------------ To enable nested KVM guests, your compute node must load the ``kvm_intel`` or ``kvm_amd`` module with ``nested=1``. You can enable @@ -462,14 +314,14 @@ content: A reboot may be required for the change to become effective. -Nested guest support in ``nova.conf`` -------------------------------------- +Nova configuration +------------------ To support nested guests, you must set your :oslo.config:option:`libvirt.cpu_mode` configuration to one of the following options: -Host pass through +Host passthrough (``host-passthrough``) In this mode, nested virtualization is automatically enabled once the KVM kernel module is loaded with nesting support. @@ -478,10 +330,11 @@ Host pass through [libvirt] cpu_mode = host-passthrough - However, do consider the other implications that `Host pass - through`_ mode has on compute functionality. + However, do consider the other implications that + :doc:`host passthrough ` mode has on compute + functionality. -Host model +Host model (``host-model``) In this mode, nested virtualization is automatically enabled once the KVM kernel module is loaded with nesting support, **if** the matching CPU model exposes the ``vmx`` feature flag to guests by @@ -495,10 +348,10 @@ Host model cpu_mode = host-model cpu_model_extra_flags = vmx - Again, consider the other implications that apply to the `Host model - (default for KVM & Qemu)`_ mode. + Again, consider the other implications that apply to the + :doc:`host model ` mode. -Custom +Custom (``custom``) In custom mode, the same considerations apply as in host-model mode, but you may *additionally* want to ensure that libvirt passes not only the ``vmx``, but also the ``pcid`` flag to its guests: @@ -510,8 +363,10 @@ Custom cpu_models = IvyBridge cpu_model_extra_flags = vmx,pcid -Nested guest support limitations --------------------------------- +More information on CPU models can be found in :doc:`/admin/cpu-models`. + +Limitations +----------- When enabling nested guests, you should be aware of (and inform your users about) certain limitations that are currently inherent to nested diff --git a/doc/source/admin/cpu-models.rst b/doc/source/admin/cpu-models.rst new file mode 100644 index 000000000000..dec450387ce3 --- /dev/null +++ b/doc/source/admin/cpu-models.rst @@ -0,0 +1,173 @@ +========== +CPU models +========== + +Nova allows you to control the guest CPU model that is exposed to instances. +Use cases include: + +* To maximize performance of instances by exposing new host CPU features to the + guest + +* To ensure a consistent default behavior across all machines, removing + reliance on system defaults. + +.. important:: + + The functionality described below is currently only supported by the + libvirt driver. + + +CPU modes +--------- + +In libvirt, the CPU is specified by providing a base CPU model name (which is a +shorthand for a set of feature flags), a set of additional feature flags, and +the topology (sockets/cores/threads). The libvirt KVM driver provides a number +of standard CPU model names. These models are defined in +``/usr/share/libvirt/cpu_map/*.xml``. You can inspect these files to determine +which models are supported by your local installation. + +Two Compute configuration options in the :oslo.config:group:`libvirt` group +of ``nova.conf`` define which type of CPU model is exposed to the hypervisor +when using KVM: :oslo.config:option:`libvirt.cpu_mode` and +:oslo.config:option:`libvirt.cpu_models`. + +The :oslo.config:option:`libvirt.cpu_mode` option can take one of the following +values: ``none``, ``host-passthrough``, ``host-model``, and ``custom``. + +See `Effective Virtual CPU configuration in Nova`__ for a recorded presentation +about this topic. + +.. __: https://www.openstack.org/videos/summits/berlin-2018/effective-virtual-cpu-configuration-in-nova + +Host model +~~~~~~~~~~ + +If :oslo.config:option:`cpu_mode=host-model `, the CPU model +in ``/usr/share/libvirt/cpu_map/*.xml`` that most closely matches the host, and +requests additional CPU flags to complete the match. This configuration +provides the maximum functionality and performance and maintains good +reliability. + +With regard to enabling and facilitating live migration between +compute nodes, you should assess whether ``host-model`` is suitable +for your compute architecture. In general, using ``host-model`` is a +safe choice if your compute node CPUs are largely identical. However, +if your compute nodes span multiple processor generations, you may be +better advised to select a ``custom`` CPU model. + +The ``host-model`` CPU model is the default for the KVM & QEMU hypervisors +(:oslo.config:option:`libvirt.virt_type`\ =``kvm``/``qemu``) + +Host passthrough +~~~~~~~~~~~~~~~~ + +If :oslo.config:option:`cpu_mode=host-passthrough `, libvirt +tells KVM to pass through the host CPU with no modifications. In comparison to +``host-model`` which simply matches feature flags, ``host-passthrough`` ensures +every last detail of the host CPU is matched. This gives the best performance, +and can be important to some apps which check low level CPU details, but it +comes at a cost with respect to migration. + +In ``host-passthrough`` mode, the guest can only be live-migrated to a +target host that matches the source host extremely closely. This +definitely includes the physical CPU model and running microcode, and +may even include the running kernel. Use this mode only if: + +* Your compute nodes have a very large degree of homogeneity + (i.e. substantially all of your compute nodes use the exact same CPU + generation and model), and you make sure to only live-migrate + between hosts with exactly matching kernel versions, *or* + +* You decide, for some reason and against established best practices, + that your compute infrastructure should not support any live + migration at all. + +Custom +~~~~~~ + +If :oslo.config:option:`cpu_mode=custom `, you can explicitly +specify an ordered list of supported named models using the +:oslo.config:option:`libvirt.cpu_models` configuration option. It is expected +that the list is ordered so that the more common and less advanced CPU models +are listed earlier. + +In selecting the ``custom`` mode, along with a +:oslo.config:option:`libvirt.cpu_models` that matches the oldest of your compute +node CPUs, you can ensure that live migration between compute nodes will always +be possible. However, you should ensure that the +:oslo.config:option:`libvirt.cpu_models` you select passes the correct CPU +feature flags to the guest. + +If you need to further tweak your CPU feature flags in the ``custom`` mode, see +`CPU feature flags`_. + +.. note:: + + If :oslo.config:option:`libvirt.cpu_models` is configured, + the CPU models in the list needs to be compatible with the host CPU. Also, if + :oslo.config:option:`libvirt.cpu_model_extra_flags` is configured, all flags + needs to be compatible with the host CPU. If incompatible CPU models or flags + are specified, nova service will raise an error and fail to start. + +None +~~~~ + +If :oslo.config:option:`cpu_mode=none `, libvirt does not +specify a CPU model. Instead, the hypervisor chooses the default model. + +The ``none`` CPU model is the default for all non-KVM.QEMU hypervisors. +(:oslo.config:option:`libvirt.virt_type`\ !=``kvm``/``qemu``) + + +CPU feature flags +----------------- + +.. versionadded:: 18.0.0 (Rocky) + +Regardless of your configured :oslo.config:option:`libvirt.cpu_mode`, it is +also possible to selectively enable additional feature flags. This can be +accomplished using the :oslo.config:option:`libvirt.cpu_model_extra_flags` +config option. For example, suppose you have configured a custom CPU model of +``IvyBridge``, which normally does not enable the ``pcid`` feature flag, but +you do want to pass ``pcid`` into your guest instances. In this case, you could +configure the following in ``nova.conf`` to enable this flag. + +.. code-block:: ini + + [libvirt] + cpu_mode = custom + cpu_models = IvyBridge + cpu_model_extra_flags = pcid + +An end user can also specify required CPU features through traits. When +specified, the libvirt driver will select the first CPU model in the +:oslo.config:option:`libvirt.cpu_models` list that can provide the requested +feature traits. If no CPU feature traits are specified then the instance will +be configured with the first CPU model in the list. + +Consider the following ``nova.conf``: + +.. code-block:: ini + + [libvirt] + cpu_mode = custom + cpu_models = Penryn,IvyBridge,Haswell,Broadwell,Skylake-Client + +These different CPU models support different feature flags and are correctly +configured in order of oldest (and therefore most widely supported) to newest. +If the user explicitly required the ``avx`` and ``avx2`` CPU features, the +latter of which is only found of Haswell-generation processors or newer, then +they could request them using the +:nova:extra-spec:`trait{group}:HW_CPU_X86_AVX` and +:nova:extra-spec:`trait{group}:HW_CPU_X86_AVX2` flavor extra specs. For +example: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property trait:HW_CPU_X86_AVX=required \ + --property trait:HW_CPU_X86_AVX2=required + +As ``Haswell`` is the first CPU model supporting both of these CPU features, +the instance would be configured with this model. diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index 39f5f0e87838..ef7fb902a8db 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -119,6 +119,7 @@ instance for these kind of workloads. sev managing-resource-providers resource-limits + cpu-models Additional guides