openstack/nova
Code Issues Proposed changes
nova/doc/source/admin/aggregates.rst
Balazs Gibizer 3c0eadae0b Reject AZ changes during aggregate add / remove host 
After this patch nova rejects the add host to aggregate API action
if the host has instances and the new aggregate for the host would
mean that these instances need to move from one AZ (even from the
default one) to another. Such AZ change is not implemented in nova
and currently leads to stuck instances.

Similarly nova will reject remove host from aggregate API action if the
host has instances and the aggregate removal would mean that the
instances need to change AZ.

Depends-On: https://review.opendev.org/c/openstack/tempest/+/821732

Change-Id: I19c4c6d34aa2cc1f32d81e8c1a52762fa3a18580
Closes-Bug: #1907775
2024-05-08 14:56:56 +02:00

403 lines
18 KiB
ReStructuredText
Raw Blame History

===============
Host aggregates
===============
Host aggregates are a mechanism for partitioning hosts in an OpenStack cloud,
or a region of an OpenStack cloud, based on arbitrary characteristics.
Examples where an administrator may want to do this include where a group of
hosts have additional hardware or performance characteristics.
Host aggregates started out as a way to use Xen hypervisor resource pools, but
have been generalized to provide a mechanism to allow administrators to assign
key-value pairs to groups of machines. Each node can have multiple aggregates,
each aggregate can have multiple key-value pairs, and the same key-value pair
can be assigned to multiple aggregates. This information can be used in the
scheduler to enable advanced scheduling, to set up Xen hypervisor resource
pools or to define logical groups for migration.
Host aggregates are not explicitly exposed to users. Instead administrators map
flavors to host aggregates. Administrators do this by setting metadata on a
host aggregate, and matching flavor extra specifications. The scheduler then
endeavors to match user requests for instances of the given flavor to a host
aggregate with the same key-value pair in its metadata. Compute nodes can be in
more than one host aggregate. Weight multipliers can be controlled on a
per-aggregate basis by setting the desired ``xxx_weight_multiplier`` aggregate
metadata.
Administrators are able to optionally expose a host aggregate as an
:term:`Availability Zone`. Availability zones are different from host
aggregates in that they are explicitly exposed to the user, and hosts can only
be in a single availability zone. Administrators can configure a default
availability zone where instances will be scheduled when the user fails to
specify one. For more information on how to do this, refer to
:doc:`/admin/availability-zones`.
.. note::
It is not allowed to move instances between Availability Zones.
If adding a host to an aggregate or removing a host from an aggregate would
cause an instance to move between Availability Zones, including moving
from or moving to the default AZ, then the operation will be rejected. The
administrator should drain the instances from the host first then the host
can be moved.
.. _config-sch-for-aggs:
Configure scheduler to support host aggregates
----------------------------------------------
One common use case for host aggregates is when you want to support scheduling
instances to a subset of compute hosts because they have a specific capability.
For example, you may want to allow users to request compute hosts that have SSD
drives if they need access to faster disk I/O, or access to compute hosts that
have GPU cards to take advantage of GPU-accelerated code.
To configure the scheduler to support host aggregates, the
:oslo.config:option:`filter_scheduler.enabled_filters` configuration option
must contain the ``AggregateInstanceExtraSpecsFilter`` in addition to the other
filters used by the scheduler. Add the following line to ``nova.conf`` on the
host that runs the ``nova-scheduler`` service to enable host aggregates
filtering, as well as the other filters that are typically enabled:
.. code-block:: ini
[filter_scheduler]
enabled_filters=...,AggregateInstanceExtraSpecsFilter
Example: Specify compute hosts with SSDs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This example configures the Compute service to enable users to request nodes
that have solid-state drives (SSDs). You create a ``fast-io`` host aggregate in
the ``nova`` availability zone and you add the ``ssd=true`` key-value pair to
the aggregate. Then, you add the ``node1``, and ``node2`` compute nodes to it.
.. code-block:: console
$ openstack aggregate create --zone nova fast-io
+-------------------+----------------------------+
| Field | Value |
+-------------------+----------------------------+
| availability_zone | nova |
| created_at | 2016-12-22T07:31:13.013466 |
| deleted | False |
| deleted_at | None |
| id | 1 |
| name | fast-io |
| updated_at | None |
+-------------------+----------------------------+
$ openstack aggregate set --property ssd=true 1
+-------------------+----------------------------+
| Field | Value |
+-------------------+----------------------------+
| availability_zone | nova |
| created_at | 2016-12-22T07:31:13.000000 |
| deleted | False |
| deleted_at | None |
| hosts | [] |
| id | 1 |
| name | fast-io |
| properties | ssd='true' |
| updated_at | None |
+-------------------+----------------------------+
$ openstack aggregate add host 1 node1
+-------------------+--------------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------------+
| availability_zone | nova |
| created_at | 2016-12-22T07:31:13.000000 |
| deleted | False |
| deleted_at | None |
| hosts | [u'node1'] |
| id | 1 |
| metadata | {u'ssd': u'true', u'availability_zone': u'nova'} |
| name | fast-io |
| updated_at | None |
+-------------------+--------------------------------------------------+
$ openstack aggregate add host 1 node2
+-------------------+--------------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------------+
| availability_zone | nova |
| created_at | 2016-12-22T07:31:13.000000 |
| deleted | False |
| deleted_at | None |
| hosts | [u'node1', u'node2'] |
| id | 1 |
| metadata | {u'ssd': u'true', u'availability_zone': u'nova'} |
| name | fast-io |
| updated_at | None |
+-------------------+--------------------------------------------------+
Use the :command:`openstack flavor create` command to create the ``ssd.large``
flavor called with an ID of 6, 8 GB of RAM, 80 GB root disk, and 4 vCPUs.
.. code-block:: console
$ openstack flavor create --id 6 --ram 8192 --disk 80 --vcpus 4 ssd.large
+----------------------------+-----------+
| Field | Value |
+----------------------------+-----------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 80 |
| id | 6 |
| name | ssd.large |
| os-flavor-access:is_public | True |
| ram | 8192 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 4 |
+----------------------------+-----------+
Once the flavor is created, specify one or more key-value pairs that match the
key-value pairs on the host aggregates with scope
``aggregate_instance_extra_specs``. In this case, that is the
``aggregate_instance_extra_specs:ssd=true`` key-value pair. Setting a
key-value pair on a flavor is done using the :command:`openstack flavor set`
command.
.. code-block:: console
$ openstack flavor set \
--property aggregate_instance_extra_specs:ssd=true ssd.large
Once it is set, you should see the ``extra_specs`` property of the
``ssd.large`` flavor populated with a key of ``ssd`` and a corresponding value
of ``true``.
.. code-block:: console
$ openstack flavor show ssd.large
+----------------------------+-------------------------------------------+
| Field | Value |
+----------------------------+-------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 80 |
| id | 6 |
| name | ssd.large |
| os-flavor-access:is_public | True |
| properties | aggregate_instance_extra_specs:ssd='true' |
| ram | 8192 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 4 |
+----------------------------+-------------------------------------------+
Now, when a user requests an instance with the ``ssd.large`` flavor,
the scheduler only considers hosts with the ``ssd=true`` key-value pair.
In this example, these are ``node1`` and ``node2``.
Aggregates in Placement
-----------------------
Aggregates also exist in placement and are not the same thing as host
aggregates in nova. These aggregates are defined (purely) as groupings of
related resource providers. Since compute nodes in nova are represented in
placement as resource providers, they can be added to a placement aggregate as
well. For example, get the UUID of the compute node using :command:`openstack
hypervisor list` and add it to an aggregate in placement using
:command:`openstack resource provider aggregate set`.
.. code-block:: console
$ openstack --os-compute-api-version=2.53 hypervisor list
+--------------------------------------+---------------------+-----------------+-----------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+---------------------+-----------------+-----------------+-------+
| 815a5634-86fb-4e1e-8824-8a631fee3e06 | node1 | QEMU | 192.168.1.123 | up |
+--------------------------------------+---------------------+-----------------+-----------------+-------+
$ openstack --os-placement-api-version=1.2 resource provider aggregate set \
--aggregate df4c74f3-d2c4-4991-b461-f1a678e1d161 \
815a5634-86fb-4e1e-8824-8a631fee3e06
Some scheduling filter operations can be performed by placement for increased
speed and efficiency.
.. note::
The nova-api service attempts (as of nova 18.0.0) to automatically mirror
the association of a compute host with an aggregate when an administrator
adds or removes a host to/from a nova host aggregate. This should alleviate
the need to manually create those association records in the placement API
using the ``openstack resource provider aggregate set`` CLI invocation.
.. _tenant-isolation-with-placement:
Tenant Isolation with Placement
-------------------------------
In order to use placement to isolate tenants, there must be placement
aggregates that match the membership and UUID of nova host aggregates that you
want to use for isolation. The same key pattern in aggregate metadata used by
the :ref:`AggregateMultiTenancyIsolation` filter controls this function, and is
enabled by setting
:oslo.config:option:`scheduler.limit_tenants_to_placement_aggregate` to
``True``.
.. code-block:: console
$ openstack --os-compute-api-version=2.53 aggregate create myagg
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| availability_zone | None |
| created_at | 2018-03-29T16:22:23.175884 |
| deleted | False |
| deleted_at | None |
| id | 4 |
| name | myagg |
| updated_at | None |
| uuid | 019e2189-31b3-49e1-aff2-b220ebd91c24 |
+-------------------+--------------------------------------+
$ openstack --os-compute-api-version=2.53 aggregate add host myagg node1
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| availability_zone | None |
| created_at | 2018-03-29T16:22:23.175884 |
| deleted | False |
| deleted_at | None |
| hosts | [u'node1'] |
| id | 4 |
| name | myagg |
| updated_at | None |
| uuid | 019e2189-31b3-49e1-aff2-b220ebd91c24 |
+-------------------+--------------------------------------+
$ openstack project list -f value | grep 'demo'
9691591f913949818a514f95286a6b90 demo
$ openstack aggregate set \
--property filter_tenant_id=9691591f913949818a514f95286a6b90 myagg
$ openstack --os-placement-api-version=1.2 resource provider aggregate set \
--aggregate 019e2189-31b3-49e1-aff2-b220ebd91c24 \
815a5634-86fb-4e1e-8824-8a631fee3e06
Note that the ``filter_tenant_id`` metadata key can be optionally suffixed
with any string for multiple tenants, such as ``filter_tenant_id3=$tenantid``.
Usage
-----
Much of the configuration of host aggregates is driven from the API or
command-line clients. For example, to create a new aggregate and add hosts to
it using the :command:`openstack` client, run:
.. code-block:: console
$ openstack aggregate create my-aggregate
$ openstack aggregate add host my-aggregate my-host
To list all aggregates and show information about a specific aggregate, run:
.. code-block:: console
$ openstack aggregate list
$ openstack aggregate show my-aggregate
To set and unset a property on the aggregate, run:
.. code-block:: console
$ openstack aggregate set --property pinned=true my-aggregrate
$ openstack aggregate unset --property pinned my-aggregate
To rename the aggregate, run:
.. code-block:: console
$ openstack aggregate set --name my-awesome-aggregate my-aggregate
To remove a host from an aggregate and delete the aggregate, run:
.. code-block:: console
$ openstack aggregate remove host my-aggregate my-host
$ openstack aggregate delete my-aggregate
For more information, refer to the :python-openstackclient-doc:`OpenStack
Client documentation <cli/command-objects/aggregate.html>`.
Configuration
-------------
In addition to CRUD operations enabled by the API and clients, the following
configuration options can be used to configure how host aggregates and the
related availability zones feature operate under the hood:
- :oslo.config:option:`default_schedule_zone`
- :oslo.config:option:`scheduler.limit_tenants_to_placement_aggregate`
- :oslo.config:option:`cinder.cross_az_attach`
Finally, as discussed previously, there are a number of host aggregate-specific
scheduler filters. These are:
- :ref:`AggregateImagePropertiesIsolation`
- :ref:`AggregateInstanceExtraSpecsFilter`
- :ref:`AggregateIoOpsFilter`
- :ref:`AggregateMultiTenancyIsolation`
- :ref:`AggregateNumInstancesFilter`
- :ref:`AggregateTypeAffinityFilter`
The following configuration options are applicable to the scheduler
configuration:
- :oslo.config:option:`cpu_allocation_ratio`
- :oslo.config:option:`ram_allocation_ratio`
- :oslo.config:option:`filter_scheduler.max_instances_per_host`
- :oslo.config:option:`filter_scheduler.aggregate_image_properties_isolation_separator`
- :oslo.config:option:`filter_scheduler.aggregate_image_properties_isolation_namespace`
.. _image-caching-aggregates:
Image Caching
-------------
Aggregates can be used as a way to target multiple compute nodes for the purpose of
requesting that images be pre-cached for performance reasons.
.. note::
`Some of the virt drivers`_ provide image caching support, which improves performance
of second-and-later boots of the same image by keeping the base image in an on-disk
cache. This avoids the need to re-download the image from Glance, which reduces
network utilization and time-to-boot latency. Image pre-caching is the act of priming
that cache with images ahead of time to improve performance of the first boot.
.. _Some of the virt drivers: https://docs.openstack.org/nova/latest/user/support-matrix.html#operation_cache_images
Assuming an aggregate called ``my-aggregate`` where two images should
be pre-cached, running the following command will initiate the
request:
.. code-block:: console
$ nova aggregate-cache-images my-aggregate image1 image2
Note that image pre-caching happens asynchronously in a best-effort
manner. The images and aggregate provided are checked by the server
when the command is run, but the compute nodes are not checked to see
if they support image caching until the process runs. Progress and
results are logged by each compute, and the process sends
``aggregate.cache_images.start``, ``aggregate.cache_images.progress``,
and ``aggregate.cache_images.end`` notifications, which may be useful
for monitoring the operation externally.
References
----------
- `Curse your bones, Availability Zones! (Openstack Summit Vancouver 2018)
<https://www.openstack.org/videos/vancouver-2018/curse-your-bones-availability-zones-1>`__
View Git Blame Copy Permalink