Implement Resource_sync commands for KB

"sync create" Sync Resources from One region to other.
"sync list"  List Sync Jobs.
"sync show"  List the details of a Sync Job.
"sync delete" Delete Sync Job(s) details from the database.
Add test-cases for the same.

Python-kingbirdclient is now a part of openstack/requirements.
https://review.openstack.org/#/c/428793/.
So updated tox.ini to pick requirements from openstack/requirements
directly rather than using install _ commands.
Also added tox_install.sh script

Closes-Bug: #1666453

Change-Id: I587070a7175ea0651def5630c2d9890a175feb0a
This commit is contained in:
Goutham Pratapa 2017-02-20 18:59:30 +05:30
parent ec51bdb8da
commit b31dedd552
9 changed files with 609 additions and 9 deletions

View File

@ -1,3 +1,15 @@
Kingbird
=========
Centralised service for multi-region OpenStack deployments.
Kingbird is an centralized OpenStack service that provides resource operation and
management across multiple OpenStack instances in a multi-region OpenStack deployment.
This service is part of the OPNFV Multisite project that intends to address
the use cases related to distributed cloud environments.
Kingbird provides features like centralized quota management, centralized view for
distributed virtual resources, global view for tenant level IP/MAC address space management,
synchronisation of ssh keys, images, flavors, etc. across regions.
=============================== ===============================
python-kingbirdclient python-kingbirdclient
=============================== ===============================
@ -8,12 +20,58 @@ This is a client library for Kingbird built on the Kingbird API. It
provides a Python API (the ``kingbirdclient`` module) and a command-line tool provides a Python API (the ``kingbirdclient`` module) and a command-line tool
(``kingbird``). (``kingbird``).
Installation
------------
First of all, clone the repo and go to the repo directory:
$ git clone https://github.com/openstack/python-kingbirdclient.git
$ cd python-kingbirdclient
Then just run:
$ pip install -e .
or
$ pip install -r requirements.txt
$ python setup.py install
Running Kingbird client
-----------------------
$ export OS_REGION_NAME=RegionOne
$ export OS_USER_DOMAIN_ID=default
$ export OS_PROJECT_NAME=<project_name>
$ export OS_IDENTITY_API_VERSION=<identity_version>
$ export OS_PASSWORD=<password>
$ export OS_AUTH_URL=http://<Keystone_host>:5000/<v3(or)v2.0>
$ export OS_USERNAME=<user_name>
$ export OS_TENANT_NAME=<tenant_name>
To make sure Kingbird client works, type:
$ kingbird quota defaults
You can see the list of available commands typing:
$ kingbird --help
Useful Links
============
* Free software: Apache license * Free software: Apache license
* Documentation: http://docs.openstack.org/developer/python-kingbirdclient * `PyPi`_ - package installation
* Source: http://git.openstack.org/cgit/openstack/python-kingbirdclient * `Launchpad project`_ - release management
* Bugs: http://bugs.launchpad.net/python-kingbirdclient * `Blueprints`_ - feature specifications
* `Bugs`_ - issue tracking
* `Source`_
* `How to Contribute`_
* `Documentation`_
Features .. _PyPi: https://pypi.python.org/pypi/python-kingbirdclient
-------- .. _Launchpad project: https://launchpad.net/python-kingbirdclient
.. _Bugs: https://bugs.launchpad.net/python-kingbirdclient
* TODO .. _Blueprints: https://blueprints.launchpad.net/python-kingbirdclient
.. _Source: http://git.openstack.org/cgit/openstack/python-kingbirdclient
.. _How to Contribute: http://docs.openstack.org/infra/manual/developers.html
.. _Documentation: http://docs.openstack.org/developer/python-kingbirdclient

View File

@ -74,6 +74,53 @@ class ResourceManager(object):
json_object['usage'][values])) json_object['usage'][values]))
return resource return resource
def resource_sync_create(self, url, data):
data = json.dumps(data)
resp = self.http_client.post(url, data)
if resp.status_code != 200:
self._raise_api_exception(resp)
json_response_key = get_json(resp)
json_object = json_response_key['job_status']
resource = []
resource.append(self.resource_class(
self, id=json_object['id'],
status=json_object['status'],
created_at=json_object['created_at']))
return resource
def _resource_sync_list(self, url):
resp = self.http_client.get(url)
if resp.status_code != 200:
self._raise_api_exception(resp)
json_response_key = get_json(resp)
json_objects = json_response_key['job_set']
resource = []
for json_object in json_objects:
resource.append(self.resource_class(
self, id=json_object['id'],
status=json_object['sync_status'],
created_at=json_object['created_at'],
updated_at=json_object['updated_at']))
return resource
def _resource_sync_detail(self, url):
resp = self.http_client.get(url)
if resp.status_code != 200:
self._raise_api_exception(resp)
json_response_key = get_json(resp)
json_objects = json_response_key['job_set']
resource = []
for json_object in json_objects:
resource.append(self.resource_class(
self, resource_name=json_object['resource'],
source_region=json_object['source_region'],
target_region=json_object['target_region'],
resource_type=json_object['resource_type'],
status=json_object['sync_status'],
created_at=json_object['created_at'],
updated_at=json_object['updated_at']))
return resource
def _delete(self, url): def _delete(self, url):
resp = self.http_client.delete(url) resp = self.http_client.delete(url)
if resp.status_code != 200: if resp.status_code != 200:

View File

@ -21,6 +21,7 @@ import osprofiler.profiler
from kingbirdclient.api import httpclient from kingbirdclient.api import httpclient
from kingbirdclient.api.v1 import quota_class_manager as qcm from kingbirdclient.api.v1 import quota_class_manager as qcm
from kingbirdclient.api.v1 import quota_manager as qm from kingbirdclient.api.v1 import quota_manager as qm
from kingbirdclient.api.v1 import sync_manager as sm
_DEFAULT_KINGBIRD_URL = "http://localhost:8118/v1.0" _DEFAULT_KINGBIRD_URL = "http://localhost:8118/v1.0"
@ -78,6 +79,7 @@ class Client(object):
# Create all resource managers # Create all resource managers
self.quota_manager = qm.quota_manager(self.http_client) self.quota_manager = qm.quota_manager(self.http_client)
self.quota_class_manager = qcm.quota_class_manager(self.http_client) self.quota_class_manager = qcm.quota_class_manager(self.http_client)
self.sync_manager = sm.sync_manager(self.http_client)
def authenticate(kingbird_url=None, username=None, def authenticate(kingbird_url=None, username=None,

View File

@ -0,0 +1,59 @@
# Copyright (c) 2017 Ericsson AB.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from kingbirdclient.api import base
class Resource(base.Resource):
resource_name = 'os-sync'
def __init__(self, manager, status, created_at, updated_at=None,
resource_type=None, target_region=None,
source_region=None, id=None, resource_name=None,):
self.manager = manager
self.id = id
self.source_region = source_region
self.target_region = target_region
self.status = status
self.created_at = created_at
self.updated_at = updated_at
self.resource_name = resource_name
self.resource_type = resource_type
class sync_manager(base.ResourceManager):
resource_class = Resource
def sync_resources(self, **kwargs):
tenant = self.http_client.project_id
data = dict()
data['resource_set'] = kwargs
url = '/%s/os-sync/' % tenant
return self.resource_sync_create(url, data)
def list_sync_jobs(self):
tenant = self.http_client.project_id
url = '/%s/os-sync/' % tenant
return self._resource_sync_list(url)
def sync_job_detail(self, job_id):
tenant = self.http_client.project_id
url = '/%s/os-sync/%s' % (tenant, job_id)
return self._resource_sync_detail(url)
def delete_sync_job(self, job_id):
tenant = self.http_client.project_id
url = '/%s/os-sync/%s' % (tenant, job_id)
return self._delete(url)

View File

@ -0,0 +1,201 @@
# Copyright (c) 2017 Ericsson AB.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from osc_lib.command import command
from kingbirdclient.commands.v1 import base
from kingbirdclient import exceptions
def format(resources=None):
columns = (
'ID',
'STATUS',
'CREATED_AT',
'UPDATED_AT',
)
if resources:
data = (
resources.id,
resources.status,
resources.created_at,
resources.updated_at,
)
else:
data = (tuple('<none>' for _ in range(len(columns))),)
return columns, data
def detail_format(resources=None):
columns = (
'RESOURCE',
'SOURCE_REGION',
'TARGET_REGION',
'RESOURCE_TYPE',
'STATUS',
'CREATED_AT',
'UPDATED_AT',
)
if resources:
data = (
resources.resource_name,
resources.source_region,
resources.target_region,
resources.resource_type,
resources.status,
resources.created_at,
resources.updated_at,
)
else:
data = (tuple('<none>' for _ in range(len(columns))),)
return columns, data
def sync_format(resources=None):
columns = (
'ID',
'STATUS',
'CREATED_AT',
)
if resources:
data = (
resources.id,
resources.status,
resources.created_at,
)
else:
data = (tuple('<none>' for _ in range(len(columns))),)
return columns, data
class ResourceSync(base.KingbirdLister):
"""Sync Resources from One region to other."""
def _get_format_function(self):
return sync_format
def get_parser(self, parsed_args):
parser = super(ResourceSync, self).get_parser(parsed_args)
parser.add_argument(
'--source',
required=True,
help='Source Region from which resources have to be synced.'
)
parser.add_argument(
'--target',
action='append',
required=True,
help='Target Region to which resources have to be synced.'
)
parser.add_argument(
'--resource_type',
required=True,
help='Type of the resource to be synced.'
)
parser.add_argument(
'--resources',
action='append',
required=True,
help='Identifier of the resource',
)
parser.add_argument(
'--force',
action='store_true',
help='Overwrites existing resources on the target regions.'
)
return parser
def _get_resources(self, parsed_args):
kingbird_client = self.app.client_manager.sync_engine
kwargs = dict()
kwargs['resource_type'] = parsed_args.resource_type
kwargs['force'] = str(parsed_args.force)
kwargs['resources'] = parsed_args.resources
kwargs['source'] = parsed_args.source
kwargs['target'] = parsed_args.target
return kingbird_client.sync_manager.sync_resources(**kwargs)
class SyncList(base.KingbirdLister):
"""List Sync Jobs."""
def _get_format_function(self):
return format
def _get_resources(self, parsed_args):
kingbird_client = self.app.client_manager.sync_engine
return kingbird_client.sync_manager.list_sync_jobs()
class SyncShow(base.KingbirdLister):
"""List the details of a Sync Job."""
def _get_format_function(self):
return detail_format
def get_parser(self, parsed_args):
parser = super(SyncShow, self).get_parser(parsed_args)
parser.add_argument(
'job_id',
help='ID of Job to view the details.'
)
return parser
def _get_resources(self, parsed_args):
job_id = parsed_args.job_id
kingbird_client = self.app.client_manager.sync_engine
return kingbird_client.sync_manager.sync_job_detail(job_id)
class SyncDelete(command.Command):
"""Delete Sync Job(s) details from the database."""
def get_parser(self, prog_name):
parser = super(SyncDelete, self).get_parser(prog_name)
parser.add_argument(
'job_id',
nargs="+",
help='ID of the job to delete entries in database.'
)
return parser
def take_action(self, parsed_args):
jobs = parsed_args.job_id
kingbird_client = self.app.client_manager.sync_engine
for job in jobs:
try:
kingbird_client.sync_manager.delete_sync_job(job)
except Exception as e:
print (e)
error_msg = "Unable to delete the entries of %s" % (job)
raise exceptions.KingbirdClientException(error_msg)

View File

@ -31,6 +31,7 @@ from osc_lib.command import command
import argparse import argparse
from kingbirdclient.commands.v1 import quota_class_manager as qcm from kingbirdclient.commands.v1 import quota_class_manager as qcm
from kingbirdclient.commands.v1 import quota_manager as qm from kingbirdclient.commands.v1 import quota_manager as qm
from kingbirdclient.commands.v1 import sync_manager as sm
LOG = logging.getLogger(__name__) LOG = logging.getLogger(__name__)
@ -403,6 +404,10 @@ class KingbirdShell(app.App):
'quota-class show': qcm.ListQuotaClass, 'quota-class show': qcm.ListQuotaClass,
'quota-class update': qcm.UpdateQuotaClass, 'quota-class update': qcm.UpdateQuotaClass,
'quota-class delete': qcm.DeleteQuotaClass, 'quota-class delete': qcm.DeleteQuotaClass,
'sync create': sm.ResourceSync,
'sync list': sm.SyncList,
'sync show': sm.SyncShow,
'sync delete': sm.SyncDelete,
} }

View File

@ -0,0 +1,172 @@
# Copyright (c) 2017 Ericsson AB.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import mock
from oslo_utils import timeutils
from oslo_utils import uuidutils
from kingbirdclient.api.v1 import sync_manager as sm
from kingbirdclient.commands.v1 import sync_manager as sync_cmd
from kingbirdclient.tests import base
TIME_NOW = timeutils.utcnow().isoformat()
ID = uuidutils.generate_uuid()
ID_1 = uuidutils.generate_uuid()
FAKE_STATUS = 'IN_PROGRESS'
FAKE_RESOURCE = 'fake_item'
FAKE_SOURCE_REGION = 'fake_region_1'
FAKE_TARGET_REGION = 'fake_region_2'
FAKE_RESOURCE_TYPE = 'fake_resource'
RESOURCE_DICT = {
'ID': ID,
'STATUS': FAKE_STATUS,
'CREATED_AT': TIME_NOW,
'UPDATED_AT': TIME_NOW
}
SYNCMANAGER = sm.Resource(mock, id=RESOURCE_DICT['ID'],
status=RESOURCE_DICT['STATUS'],
created_at=RESOURCE_DICT['CREATED_AT'],
updated_at=RESOURCE_DICT['UPDATED_AT'])
DETAIL_RESOURCE_DICT = {
'RESOURCE': FAKE_RESOURCE,
'SOURCE_REGION': FAKE_SOURCE_REGION,
'TARGET_REGION': FAKE_TARGET_REGION,
'RESOURCE_TYPE': FAKE_RESOURCE_TYPE,
'STATUS': FAKE_STATUS,
'CREATED_AT': TIME_NOW,
'UPDATED_AT': TIME_NOW
}
DETAIL_RESOURCEMANAGER = sm.Resource(
mock, resource_name=DETAIL_RESOURCE_DICT['RESOURCE'],
source_region=DETAIL_RESOURCE_DICT['SOURCE_REGION'],
target_region=DETAIL_RESOURCE_DICT['TARGET_REGION'],
resource_type=DETAIL_RESOURCE_DICT['RESOURCE_TYPE'],
status=DETAIL_RESOURCE_DICT['STATUS'],
created_at=DETAIL_RESOURCE_DICT['CREATED_AT'],
updated_at=DETAIL_RESOURCE_DICT['UPDATED_AT'])
SYNC_RESOURCEMANAGER = sm.Resource(mock, id=RESOURCE_DICT['ID'],
status=RESOURCE_DICT['STATUS'],
created_at=RESOURCE_DICT['CREATED_AT'])
class TestCLISyncManagerV1(base.BaseCommandTest):
def test_sync_jobs_list(self):
self.client.sync_manager.list_sync_jobs.return_value = [SYNCMANAGER]
actual_call = self.call(sync_cmd.SyncList)
self.assertEqual([(ID, FAKE_STATUS, TIME_NOW, TIME_NOW)],
actual_call[1])
def test_negative_sync_jobs_list(self):
self.client.sync_manager.list_sync_jobs.return_value = []
actual_call = self.call(sync_cmd.SyncList)
self.assertEqual((('<none>', '<none>', '<none>', '<none>'),),
actual_call[1])
def test_delete_sync_job_with_job_id(self):
self.call(sync_cmd.SyncDelete, app_args=[ID])
self.client.sync_manager.delete_sync_job.\
assert_called_once_with(ID)
def test_delete_multiple_sync_jobs(self):
self.call(sync_cmd.SyncDelete, app_args=[ID, ID_1])
self.assertEqual(2,
self.client.sync_manager.delete_sync_job.call_count)
def test_delete_sync_job_without_job_id(self):
self.assertRaises(SystemExit, self.call,
sync_cmd.SyncDelete, app_args=[])
def test_detail_sync_job_with_job_id(self):
self.client.sync_manager.sync_job_detail.\
return_value = [DETAIL_RESOURCEMANAGER]
actual_call = self.call(sync_cmd.SyncShow, app_args=[ID])
self.assertEqual([(FAKE_RESOURCE, FAKE_SOURCE_REGION,
FAKE_TARGET_REGION, FAKE_RESOURCE_TYPE,
FAKE_STATUS, TIME_NOW, TIME_NOW)], actual_call[1])
def test_detail_sync_job_negative(self):
self.client.sync_manager.sync_job_detail.return_value = []
actual_call = self.call(sync_cmd.SyncShow, app_args=[ID])
self.assertEqual((('<none>', '<none>', '<none>', '<none>',
'<none>', '<none>', '<none>'),), actual_call[1])
def test_detail_sync_job_without_job_id(self):
self.assertRaises(SystemExit, self.call,
sync_cmd.SyncShow, app_args=[])
def test_resource_sync_without_force(self):
self.client.sync_manager.sync_resources.\
return_value = [SYNC_RESOURCEMANAGER]
actual_call = self.call(
sync_cmd.ResourceSync, app_args=[
'--resource_type', FAKE_RESOURCE_TYPE,
'--resources', FAKE_RESOURCE,
'--source', FAKE_SOURCE_REGION, '--target',
FAKE_TARGET_REGION])
self.assertEqual([(ID, FAKE_STATUS, TIME_NOW)], actual_call[1])
def test_resource_sync_without_resources(self):
self.client.sync_manager.sync_resources.\
return_value = [SYNC_RESOURCEMANAGER]
self.assertRaises(
SystemExit, self.call, sync_cmd.ResourceSync, app_args=[
'--resource_type', FAKE_RESOURCE_TYPE,
'--source', FAKE_SOURCE_REGION,
'--target', FAKE_TARGET_REGION])
def test_resource_sync_without_resource_type(self):
self.client.sync_manager.sync_resources.\
return_value = [SYNC_RESOURCEMANAGER]
self.assertRaises(
SystemExit, self.call, sync_cmd.ResourceSync, app_args=[
'--resources', FAKE_RESOURCE,
'--source', FAKE_SOURCE_REGION,
'--target', FAKE_TARGET_REGION])
def test_resource_sync_without_source_region(self):
self.client.sync_manager.sync_resources.\
return_value = [SYNC_RESOURCEMANAGER]
self.assertRaises(
SystemExit, self.call, sync_cmd.ResourceSync, app_args=[
'--resource_type', FAKE_RESOURCE_TYPE,
'--resources', FAKE_RESOURCE,
'--target', FAKE_TARGET_REGION])
def test_resource_sync_without_target_region(self):
self.client.sync_manager.sync_resources.\
return_value = [SYNC_RESOURCEMANAGER]
self.assertRaises(
SystemExit, self.call, sync_cmd.ResourceSync, app_args=[
'--resource_type', FAKE_RESOURCE_TYPE,
'--resources', FAKE_RESOURCE,
'--source', FAKE_SOURCE_REGION])
def test_resource_sync_with_force(self):
self.client.sync_manager.sync_resources.\
return_value = [SYNC_RESOURCEMANAGER]
actual_call = self.call(
sync_cmd.ResourceSync, app_args=[
'--resource_type', FAKE_RESOURCE_TYPE,
'--resources', FAKE_RESOURCE,
'--source', FAKE_SOURCE_REGION,
'--target', FAKE_TARGET_REGION,
'--force'])
self.assertEqual([(ID, FAKE_STATUS, TIME_NOW)], actual_call[1])

56
tools/tox_install.sh Executable file
View File

@ -0,0 +1,56 @@
#!/usr/bin/env bash
# Upper constraint file contains kingbirdclient version pin that is in
# conflict with installing kingbird from source. We should replace
# the version pin in the constraints file before applying it for from-source
# installation.
ZUUL_CLONER=/usr/zuul-env/bin/zuul-cloner
BRANCH_NAME=master
MODULE_NAME=python-kingbirdclient
requirements_installed=$(echo "import openstack_requirements" | python 2>/dev/null ; echo $?)
set -e
CONSTRAINTS_FILE=$1
shift
install_cmd="pip install"
mydir=$(mktemp -dt "$MODULE_NAME-tox_install-XXXXXXX")
trap "rm -rf $mydir" EXIT
localfile=$mydir/upper-constraints.txt
if [[ $CONSTRAINTS_FILE != http* ]]; then
CONSTRAINTS_FILE=file://$CONSTRAINTS_FILE
fi
curl $CONSTRAINTS_FILE -k -o $localfile
install_cmd="$install_cmd -c$localfile"
if [ $requirements_installed -eq 0 ]; then
echo "ALREADY INSTALLED" > /tmp/tox_install.txt
echo "Requirements already installed; using existing package"
elif [ -x "$ZUUL_CLONER" ]; then
echo "ZUUL CLONER" > /tmp/tox_install.txt
pushd $mydir
$ZUUL_CLONER --cache-dir \
/opt/git \
--branch $BRANCH_NAME \
git://git.openstack.org \
openstack/requirements
cd openstack/requirements
$install_cmd -e .
popd
else
echo "PIP HARDCODE" > /tmp/tox_install.txt
if [ -z "$REQUIREMENTS_PIP_LOCATION" ]; then
REQUIREMENTS_PIP_LOCATION="git+https://git.openstack.org/openstack/requirements@$BRANCH_NAME#egg=requirements"
fi
$install_cmd -U -e ${REQUIREMENTS_PIP_LOCATION}
fi
# This is the main purpose of the script: Allow local installation of
# the current repo. It is listed in constraints file and thus any
# install will be constrained and we need to unconstrain it.
edit-constraints $localfile -- $MODULE_NAME "-e file://$PWD#egg=$MODULE_NAME"
$install_cmd -U $*
exit $?

View File

@ -5,7 +5,7 @@ skipsdist = True
[testenv] [testenv]
usedevelop = True usedevelop = True
install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages} install_command = {toxinidir}/tools/tox_install.sh {env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}
setenv = VIRTUAL_ENV={envdir} setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt -r{toxinidir}/test-requirements.txt
@ -38,4 +38,4 @@ builtins = _
exclude=.venv,.git,.tox,dist,doc,*lib/python*,*openstack/common*,*egg,build exclude=.venv,.git,.tox,dist,doc,*lib/python*,*openstack/common*,*egg,build
[hacking] [hacking]
import_exceptions = kingbirdclient.common.i18n import_exceptions = kingbirdclient.common.i18n