Dynamic pollsters: Append relative url_path to URLs

When generating OpenStack API query URLs for dynamic pollsters,
Ceilometer uses urllib.parse.urljoin to append url_path to the
endpoint URL.

If an API endpoint URL returned by Keystone does not have a
trailing slash and the specified url_path value is a relative
path (e.g. does not start with /), urljoin will replace the
final segment of the base URL instead of appending url_path to it.

This breaks generating links in dynamic pollsters for OpenStack services
if the endpoint URL in the Keystone service catalog does not have a trailing
slash. This is particularly relevant for OpenStack services that incorporate
the project ID into the API endpoint URL, such as Trove as shown below.

>>> urljoin("http://example.com:8779/v1.0/12345678", "mgmt/instances")
'http://example.com:8779/v1.0/mgmt/instances'

If there is a trailing slash on the end of the base URL,
url_path will be appended to the end of the URL as expected.

>>> urljoin("http://example.com:8779/v1.0/12345678/", "mgmt/instances")
'http://example.com:8779/v1.0/12345678/mgmt/instances'

This commit changes the behaviour of the dynamic pollster URL joins
by making sure there is a trailing slash on the endpoint URLs when
appending the defined url_path to them.

This may break existing dynamic pollster configurations that rely
on this behaviour, but given that most OpenStack services probably
wouldn't, there shouldn't be too many changes that would be required.

Change-Id: I367d299df85676472c7e1a922d08971810f2ba9f

ceilometer/polling/dynamic_pollster.py

@@ -933,7 +933,10 @@ class PollsterSampleGatherer(object):
             LOG.debug("Cannot eval path [%s] with params [%s],"
                       " using [%s] instead.",
                       url_path, params, url_path)
-        return urlparse.urljoin(endpoint, url_path)
+        return urlparse.urljoin((endpoint
+                                 if endpoint.endswith("/")
+                                 else (endpoint + "/")),
+                                url_path)
 
     def retrieve_entries_from_response(self, response_json, definitions):
         if isinstance(response_json, list):
@@ -1091,7 +1094,10 @@ class NonOpenStackApisSamplesGatherer(PollsterSampleGatherer):
         endpoint = self.definitions.configurations['url_path']
         if endpoint == url_path:
             return url_path
-        return urlparse.urljoin(endpoint, url_path)
+        return urlparse.urljoin((endpoint
+                                 if endpoint.endswith("/")
+                                 else (endpoint + "/")),
+                                url_path)
 
     def generate_new_attributes_in_sample(
             self, sample, attribute_key, new_attribute_key):

ceilometer/tests/unit/polling/test_dynamic_pollster.py

@@ -1713,11 +1713,29 @@ class TestDynamicPollster(base.BaseTestCase):
                          list(map(lambda s: s.resource_metadata["flavor.ram"],
                                   samples)))
 
-    def test_get_request_linked_samples_url_no_next_sample(self):
+    def test_get_request_linked_samples_url_endpoint_no_trailing_slash(self):
         pollster = dynamic_pollster.DynamicPollster(
             self.pollster_definition_only_required_fields)
 
-        base_url = "http://test.com/something_that_we_do_not_care"
+        base_url = (
+            "http://test.com:8779/v1.0/1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a"
+        )
+        expected_url = urlparse.urljoin(
+            base_url + "/", self.pollster_definition_only_required_fields[
+                'url_path'])
+
+        kwargs = {'resource': base_url}
+        url = pollster.definitions.sample_gatherer\
+            .get_request_linked_samples_url(
+                kwargs, pollster.definitions.configurations)
+
+        self.assertEqual(expected_url, url)
+
+    def test_get_request_linked_samples_url_endpoint_trailing_slash(self):
+        pollster = dynamic_pollster.DynamicPollster(
+            self.pollster_definition_only_required_fields)
+
+        base_url = "http://test.com:9511/v1/"
         expected_url = urlparse.urljoin(
             base_url, self.pollster_definition_only_required_fields[
                 'url_path'])

releasenotes/notes/dynamic-pollster-url-joins-6cdb01c4015976f7.yaml

@@ -0,0 +1,14 @@
+---
+upgrade:
+  - |
+    When using dynamic pollsters to query OpenStack APIs, if the endpoint URL
+    returned by Keystone does not have a trailing slash and ``url_path`` is
+    a relative path, the ``url_path`` configured in the dynamic pollster would
+    replace sections of the endpoint URL instead of being appended to the end
+    of the URL. This behaviour has now been changed so that ``url_path``
+    values that do not start with a ``/`` are always appended to the end of
+    endpoint URLs.
+    This change may require existing dynamic pollsters that rely on this
+    behaviour to be changed, but this allows dynamic pollsters to be added
+    for OpenStack services that append the active project ID to the API
+    endpoint URL (e.g. Trove).