Enhance compute capability filter to check multi-level

Host state capabilities include multi-level information, like
cpu features are presented in capabilities:cpu_info:features, cpu
core number in capabilities:cpu_info:topology:cores.
    Multi-level capability check maybe helpful in some situation, like
user want to make sure host support SSE3 instructions or AES-NI, or
user want to make sure the host has >4 cores for SMP instance. Current
capability filter does not support these requirements.
    To support multi-level check, also add scope to capability filter,
so, now capability filter will handle no-scope extra_specs, or scope-format
as "capabilities:xxx:xxx" etc.

    Related to bp hw-feature-to-vm

Change-Id: Id714f99b4babbdb328b8e47f5a270be517b528d2
Signed-off-by: Yunhong, Jiang <yunhong.jiang@intel.com>

doc/source/devref/filter_scheduler.rst

@@ -32,10 +32,16 @@ There are some standard filter classes to use (:mod:`nova.scheduler.filters`):
   image properties contained in the instance.
 * |AvailabilityZoneFilter| - filters hosts by availability zone. It passes
   hosts matching the availability zone specified in the instance properties.
-* |ComputeCapabilityFilter| - checks that the capabilities provided by the
+* |ComputeCapabilitiesFilter| - checks that the capabilities provided by the
   host compute service satisfy any extra specifications associated with the
-  instance type (that have no scope, see |TrustedFilter| for details).  It
-  passes hosts that can create the specified instance type.
+  instance type.  It passes hosts that can create the specified instance type.
+
+  The extra specifications can have a scope at the beginning of the key string
+  of a key/value pair. The scope format is "scope:key" and can be nested,
+  i.e. key_string := scope:key_string. Example like "capabilities:cpu_info:
+  features" is valid scope format. A key string without any ':' is non-scope
+  format. Each filter defines it's valid scope, and not all filters accept
+  non-scope format.
 
   The extra specifications can have an operator at the beginning of the value
   string of a key/value pair. If there is no operator specified, then a
@@ -63,7 +69,7 @@ There are some standard filter classes to use (:mod:`nova.scheduler.filters`):
   satisfies any extra specifications associated with the instance type (that
   have no scope).  It passes hosts that can create the specified instance type.
   The extra specifications can have the same operators as
-  |ComputeCapabilityFilter|.
+  |ComputeCapabilitiesFilter|.
 * |ComputeFilter| - passes all hosts that are operational and enabled.
 * |CoreFilter| - filters based on CPU core utilization. It passes hosts with
   sufficient number of CPU cores.
@@ -128,11 +134,14 @@ hypervisor_type=qemu`
 Only hosts that satisfy these requirements will pass the
 |ImagePropertiesFilter|.
 
-|ComputeCapabilitesFilter| checks if the host satisfies any 'extra specs'
-specified on the instance type.  The 'extra specs' can contain key/value pairs,
-and the |ComputeCapabilitiesFilter| will only pass hosts whose capabilities
-satisfy the requested specifications.  All hosts are passed if no 'extra specs'
-are specified.
+|ComputeCapabilitiesFilter| checks if the host satisfies any 'extra specs'
+specified on the instance type.  The 'extra specs' can contain key/value pairs.
+The key for the filter is either non-scope format (i.e. no ':' contained), or
+scope format in capabilities scope (i.e. 'capabilities:xxx:yyy'). One example
+of capabilities scope is "capabilities:cpu_info:features", which will match
+host's cpu features capabilities. The |ComputeCapabilitiesFilter| will only
+pass hosts whose capabilities satisfy the requested specifications.  All hosts
+are passed if no 'extra specs' are specified.
 
 |ComputeFilter| is quite simple and passes any host whose compute service is
 enabled and operational.
@@ -179,9 +188,9 @@ The |RetryFilter| filters hosts that have already been attempted for scheduling.
 It only passes hosts that have not been previously attempted.
 
 The |TrustedFilter| filters hosts based on their trust.  Only passes hosts
-that match the trust requested in the `extra_specs' for the flavor.  The key
-for this filter is `trust:trusted_host', where `trust' is the scope of the
-key and `trusted_host' is the actual key value'.
+that match the trust requested in the `extra_specs' for the flavor. The key
+for this filter must be scope format as `trust:trusted_host', where `trust'
+is the scope of the key and `trusted_host' is the actual key value.
 The value of this pair (`trusted'/`untrusted') must match the
 integrity of a host (obtained from the Attestation service) before it is
 passed by the |TrustedFilter|.
@@ -198,11 +207,11 @@ The default values for these settings in nova.conf are:
 ::
 
     --scheduler_available_filters=nova.scheduler.filters.standard_filters
-    --scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilityFilter,ImagePropertiesFilter
+    --scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter
 
 With this configuration, all filters in `nova.scheduler.filters`
 would be available, and by default the |RamFilter|, |ComputeFilter|,
-|AvailabilityZoneFilter|, |ComputeCapabilityFilter|, and
+|AvailabilityZoneFilter|, |ComputeCapabilitiesFilter|, and
 |ImagePropertiesFilter| would be used.
 
 If you want to create **your own filter** you just need to inherit from

nova/scheduler/filters/compute_capabilities_filter.py

@@ -31,11 +31,20 @@ class ComputeCapabilitiesFilter(filters.BaseHostFilter):
             return True
 
         for key, req in instance_type['extra_specs'].iteritems():
-            # NOTE(jogo) any key containing a scope (scope is terminated
-            # by a `:') will be ignored by this filter. (bug 1039386)
-            if key.count(':'):
+            # Either not scope format, or in capabilities scope
+            scope = key.split(':')
+            if len(scope) > 1 and scope[0] != "capabilities":
                 continue
-            cap = capabilities.get(key, None)
+            elif scope[0] == "capabilities":
+                del scope[0]
+            cap = capabilities
+            for index in range(0, len(scope)):
+                try:
+                    cap = cap.get(scope[index], None)
+                except AttributeError:
+                    return False
+                if cap is None:
+                    return False
             if not extra_specs_ops.match(cap, req):
                 return False
         return True

nova/tests/scheduler/test_host_filters.py

@@ -742,6 +742,27 @@ class HostFiltersTestCase(test.TestCase):
             especs={'opt1': '1', 'opt2': '222', 'trust:trusted_host': 'true'},
             passes=False)
 
+    def test_compute_filter_pass_extra_specs_simple_with_scope(self):
+        self._do_test_compute_filter_extra_specs(
+            ecaps={'opt1': '1', 'opt2': '2'},
+            especs={'capabilities:opt1': '1',
+                    'trust:trusted_host': 'true'},
+            passes=True)
+
+    def test_compute_filter_extra_specs_simple_with_wrong_scope(self):
+        self._do_test_compute_filter_extra_specs(
+            ecaps={'opt1': '1', 'opt2': '2'},
+            especs={'wrong_scope:opt1': '1',
+                    'trust:trusted_host': 'true'},
+            passes=True)
+
+    def test_compute_filter_extra_specs_pass_multi_level_with_scope(self):
+        self._do_test_compute_filter_extra_specs(
+            ecaps={'opt1': {'a': '1', 'b': {'aa': '2'}}, 'opt2': '2'},
+            especs={'opt1:a': '1', 'capabilities:opt1:b:aa': '2',
+                    'trust:trusted_host': 'true'},
+            passes=True)
+
     def test_aggregate_filter_passes_no_extra_specs(self):
         self._stub_service_is_up(True)
         filt_cls = self.class_map['AggregateInstanceExtraSpecsFilter']()