Merge pull request #10 from denismakogon/update-to-latest-iron-functions-api

Massive update
This commit is contained in:
Derek Schultz 2016-12-06 10:16:11 -07:00 committed by GitHub
commit fdc9fd3935
31 changed files with 528 additions and 219 deletions

View File

@ -11,5 +11,5 @@ RUN pip3 install -r /code/requirements.txt
RUN python3 /code/setup.py install RUN python3 /code/setup.py install
ENTRYPOINT ["python3", "/code/laos/service/laos_api.py"] ENTRYPOINT ["python3", "/code/service/laos_api.py"]
EXPOSE 10001 EXPOSE 10001

View File

@ -1,6 +1,6 @@
LAOS_HOST=0.0.0.0 LAOS_HOST=0.0.0.0
LAOS_PORT=10001 LAOS_PORT=10001
LAOS_DB=mysql://root:ubuntu@192.168.0.120/functions LAOS_DB=mysql+pymysql://root:ubuntu@192.168.0.120/functions
KEYSTONE_ENDPOINT=http://192.168.0.120:5000/v3 KEYSTONE_ENDPOINT=http://192.168.0.120:5000/v3
FUNCTIONS_URL=http://192.168.0.120:8080/v1 FUNCTIONS_URL=http://192.168.0.120:8080/v1
LAOS_LOG_LEVEL=INFO LAOS_LOG_LEVEL=INFO

105
README.md
View File

@ -168,111 +168,6 @@ Once server is launched you can navigate to:
to see recent API docs to see recent API docs
Testing (general information)
-----------------------------
In order to run tests you need to install `Tox`:
$ pip install tox
Also, you will need a running MySQL instance with the database migrations applied from the previous step.
Tests are dependent on pre-created MySQL database for persistence.
Please set env var
$ export TEST_DB_URI=mysql://<your-user>:<your-user-password>@<mysql-host>:<mysql-port>/<functions-db>
Testing: PEP8
-------------
In order to run `PEP8` style checks run following command:
$ tox -e pep8
Testing: Functional
-------------------
In order to run `functional` tests run following command:
$ tox -e py35-functional
Pros:
* lightweight (controllers and DB models testing)
* no OpenStack required
* no IronFunctions required
Cons:
* MySQL server required
* OpenStack authentication is not tested
* IronFunctions API stubbed with fake implementation
Testing: Integration
--------------------
Integration tests are dependent on following env variables:
* TEST_DB_URI - similar to functional tests, database endpoint
* FUNCTIONS_API_URL - IronFunctions API URL (default value - `http://localhost:8080/v1`)
* OS_AUTH_URL - OpenStack Identity endpoint
* OS_PROJECT_NAME - OpenStack user-specific project name
* OS_USERNAME - OpenStack user name
* OS_PASSWORD - OpenStack user user password
To run tests use following command:
export TEST_DB_URI=mysql://<your-user>:<your-user-password>@<mysql-host>:<mysql-port>/<functions-db>
export FUNCTIONS_API_URL=<functions-api-protocol>://<functions-host>:<functions-port>/<functions-api-version>
export OS_AUTH_URL=<identity-api-protocol>://<identity-host>:<identity-port>/<identity-api-version>
export OS_PROJECT_NAME=<project-name>
export OS_USERNAME=<project-name>
export OS_PASSWORD=<project-name>
tox -epy35-integration
Testing: Coverage regression
----------------------------
In order to build quality software it is necessary to keep test coverage at its highest point.
So, as part of `Tox` testing new check was added - functional test coverage regression.
In order to run it use following command:
$ tox -e py35-functional-regression
3rd party bugs to resolve
-------------------------
IronFunctions:
* https://github.com/iron-io/functions/issues/298
* https://github.com/iron-io/functions/issues/296
* https://github.com/iron-io/functions/issues/275
* https://github.com/iron-io/functions/issues/274
TODOs
-----
Swagger doc:
* Make swagger doc more explicit on HTTP POST/UPDATE body content
* HTTP headers requests
IronFunctions:
* Support app deletion in IronFunctions
* Support tasks listing/showing
Laos:
* Tests: integration, functional, units
* Better logging coverage
Python Functions client:
* Support logging instance passing in [function-python](https://github.com/iron-io/functions_python)
* python-laosclient (ReST API client and CLI tool)
* App writing examples
Contacts Contacts
-------- --------

View File

@ -17,9 +17,9 @@ from aiohttp import web
from aioservice.http import controller from aioservice.http import controller
from aioservice.http import requests from aioservice.http import requests
from laos.api.views import app as app_view from ...common import config
from laos.common import config from ...models import app as app_model
from laos.models import app as app_model from ..views import app as app_view
class AppV1Controller(controller.ServiceController): class AppV1Controller(controller.ServiceController):
@ -159,22 +159,58 @@ class AppV1Controller(controller.ServiceController):
}, },
status=200 status=200
) )
# TODO(denismakogon): disabled until iron-io/functions/pull/259
# @requests.api_action(method='PUT', route='{project_id}/apps/{app}')
# @requests.api_action(method='PUT', route='{project_id}/apps/{app}') async def update(self, request, **kwargs):
# async def update(self, request, **kwargs): """
# log = config.Config.config_instance().logger ---
# project_id = request.match_info.get('project_id') description: Updating project-scoped app
# app = request.match_info.get('app') tags:
# data = await request.json() - Apps
# log.info("Updating an app {} for project: {} with data {}" produces:
# .format(app, project_id, str(data))) - application/json
# return web.json_response( responses:
# data={ "200":
# "app": {} description: successful operation. Return "app" JSON
# }, "401":
# status=200 description: Not authorized.
# ) "404":
description: App not found
"""
project_id = request.match_info.get('project_id')
app_name = request.match_info.get('app')
data = await request.json()
if not (await app_model.Apps.exists(app_name, project_id)):
return web.json_response(data={
"error": {
"message": "App {0} not found".format(app_name),
}
}, status=404)
c = config.Config.config_instance()
fnclient = c.functions_client
try:
fn_app = await fnclient.apps.update(
app_name, loop=c.event_loop, **data)
except Exception as ex:
return web.json_response(data={
"error": {
"message": getattr(ex, "reason", str(ex)),
}
}, status=getattr(ex, "status", 500))
stored_app = (await app_model.Apps.find_by(
project_id=project_id, name=app_name)).pop()
c.logger.info("Updating app {} for project: {} with data {}"
.format(app_name, project_id, str(data)))
return web.json_response(
data={
"app": app_view.AppView(stored_app, fn_app).view(),
"message": "App successfully updated"
},
status=200
)
@requests.api_action(method='DELETE', route='{project_id}/apps/{app}') @requests.api_action(method='DELETE', route='{project_id}/apps/{app}')
async def delete(self, request, **kwargs): async def delete(self, request, **kwargs):
@ -217,9 +253,8 @@ class AppV1Controller(controller.ServiceController):
await app_model.Apps.delete( await app_model.Apps.delete(
project_id=project_id, name=app) project_id=project_id, name=app)
# TODO(denismakogon): enable DELETE to IronFunctions when once await fnclient.apps.delete(app, loop=c.event_loop)
# https://github.com/iron-io/functions/issues/274 implemented
# fn_app = await fnclient.apps.delete(app, loop=c.event_loop)
return web.json_response( return web.json_response(
data={ data={
"message": "App successfully deleted", "message": "App successfully deleted",

View File

@ -19,9 +19,9 @@ from aiohttp import web
from aioservice.http import controller from aioservice.http import controller
from aioservice.http import requests from aioservice.http import requests
from laos.api.views import app as app_view from ...common import config
from laos.common import config from ...models import app as app_model
from laos.models import app as app_model from ..views import app as app_view
class AppRouteV1Controller(controller.ServiceController): class AppRouteV1Controller(controller.ServiceController):
@ -78,6 +78,7 @@ class AppRouteV1Controller(controller.ServiceController):
return web.json_response(data={ return web.json_response(data={
"routes": app_view.AppRouteView(api_url, "routes": app_view.AppRouteView(api_url,
project_id, project_id,
app,
fn_app_routes).view(), fn_app_routes).view(),
"message": "Successfully loaded app routes", "message": "Successfully loaded app routes",
}, status=200) }, status=200)
@ -166,7 +167,7 @@ class AppRouteV1Controller(controller.ServiceController):
**data, loop=c.event_loop)) **data, loop=c.event_loop))
stored_route = await app_model.Routes( stored_route = await app_model.Routes(
app_name=new_fn_route.appname, app_name=app,
project_id=project_id, project_id=project_id,
path=new_fn_route.path, path=new_fn_route.path,
is_public=is_public).save() is_public=is_public).save()
@ -178,7 +179,7 @@ class AppRouteV1Controller(controller.ServiceController):
setattr(new_fn_route, "is_public", stored_route.public) setattr(new_fn_route, "is_public", stored_route.public)
view = app_view.AppRouteView( view = app_view.AppRouteView(
api_url, project_id, [new_fn_route]).view_one() api_url, project_id, app, [new_fn_route]).view_one()
return web.json_response(data={ return web.json_response(data={
"route": view, "route": view,
@ -244,10 +245,76 @@ class AppRouteV1Controller(controller.ServiceController):
return web.json_response(data={ return web.json_response(data={
"route": app_view.AppRouteView(api_url, "route": app_view.AppRouteView(api_url,
project_id, project_id,
app,
[route]).view_one(), [route]).view_one(),
"message": "App route successfully loaded" "message": "App route successfully loaded"
}, status=200) }, status=200)
@requests.api_action(
method='PUT', route='{project_id}/apps/{app}/routes/{route}')
async def update(self, request, **kwargs):
"""
---
description: Updating project-scoped app route
tags:
- Routes
produces:
- application/json
responses:
"200":
description: Successful operation. Return empty JSON
"401":
description: Not authorized.
"404":
description: App does not exist
"404":
description: App route does not exist
"""
c = config.Config.config_instance()
log, fnclient = c.logger, c.functions_client
project_id = request.match_info.get('project_id')
app = request.match_info.get('app')
path = request.match_info.get('route')
data = await request.json()
log.info("Deleting route {} in app {} for project: {}"
.format(path, app, project_id))
if not (await app_model.Apps.exists(app, project_id)):
return web.json_response(data={
"error": {
"message": "App {0} not found".format(app),
}
}, status=404)
try:
fn_app = await fnclient.apps.show(app, loop=c.event_loop)
await fn_app.routes.show("/{}".format(path), loop=c.event_loop)
route = await fn_app.routes.update(
"/{}".format(path), loop=c.event_loop, **data)
except Exception as ex:
return web.json_response(data={
"error": {
"message": getattr(ex, "reason", str(ex)),
}
}, status=getattr(ex, "status", 500))
api_url = "{}://{}".format(request.scheme, request.host)
stored_route = (await app_model.Routes.find_by(
app_name=app,
project_id=project_id,
path=route.path)).pop()
setattr(route, "is_public", stored_route.public)
return web.json_response(data={
"route": app_view.AppRouteView(api_url,
project_id,
app,
[route]).view_one(),
"message": "Route successfully updated",
}, status=200)
@requests.api_action( @requests.api_action(
method='DELETE', route='{project_id}/apps/{app}/routes/{route}') method='DELETE', route='{project_id}/apps/{app}/routes/{route}')
async def delete(self, request, **kwargs): async def delete(self, request, **kwargs):

View File

@ -17,7 +17,7 @@ from aiohttp import web
from aioservice.http import controller from aioservice.http import controller
from aioservice.http import requests from aioservice.http import requests
from laos.common import config from ...common import config
class RunnableMixin(object): class RunnableMixin(object):

View File

@ -14,12 +14,12 @@
from aiohttp import web from aiohttp import web
# from laos.models import app as app_model # from ...models import app as app_model
from aioservice.http import controller from aioservice.http import controller
from aioservice.http import requests from aioservice.http import requests
# from laos.common import config # from ...common import config
# TODO(denismakogon): disabled until # TODO(denismakogon): disabled until

View File

@ -18,7 +18,7 @@ from keystoneclient import client
from aiohttp import web from aiohttp import web
from laos.common import config from ...common import config
async def auth_through_token(app: web.Application, handler): async def auth_through_token(app: web.Application, handler):

View File

@ -33,10 +33,11 @@ class AppView(object):
class AppRouteView(object): class AppRouteView(object):
def __init__(self, api_url, project_id, fn_app_routes): def __init__(self, api_url, project_id, app_name, fn_app_routes):
self.routes = fn_app_routes self.routes = fn_app_routes
self.api_url = api_url self.api_url = api_url
self.project_id = project_id self.project_id = project_id
self.app_name = app_name
def view_one(self): def view_one(self):
return self.view().pop() return self.view().pop()
@ -47,15 +48,23 @@ class AppRouteView(object):
if not route.is_public: if not route.is_public:
path = ("{}/v1/r/{}/{}{}".format( path = ("{}/v1/r/{}/{}{}".format(
self.api_url, self.project_id, self.api_url, self.project_id,
route.appname, route.path)) self.app_name, route.path))
else: else:
path = ("{}/r/{}{}".format( path = ("{}/r/{}{}".format(
self.api_url, route.appname, route.path)) self.api_url, self.app_name, route.path))
view.append({ one = {
"path": path, "path": path,
"type": route.type, "type": route.type,
"memory": route.memory,
"image": route.image, "image": route.image,
"is_public": route.is_public, "is_public": route.is_public,
}) }
# temporary solution for
# https://github.com/iron-io/functions/issues/382
if hasattr(route, "memory"):
one.update(memory=route.memory)
if hasattr(route, "timeout"):
one.update(timeout=route.timeout)
if hasattr(route, "max_concurrency"):
one.update(timeout=route.max_concurrency)
view.append(one)
return view return view

View File

@ -16,7 +16,7 @@
import aiomysql import aiomysql
import asyncio import asyncio
from laos.common import utils from . import utils
from functionsclient.v1 import client from functionsclient.v1 import client

View File

@ -16,7 +16,7 @@ import datetime
import logging import logging
import sys import sys
from laos.common import utils from . import utils
def common_logger_setup( def common_logger_setup(

View File

@ -15,7 +15,7 @@
import datetime import datetime
import uuid import uuid
from laos.common import config from . import config
class BaseDatabaseModel(object): class BaseDatabaseModel(object):

View File

@ -12,7 +12,7 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
from laos.common import persistence from ..common import persistence
class Apps(persistence.BaseDatabaseModel): class Apps(persistence.BaseDatabaseModel):

View File

@ -29,8 +29,9 @@ class AppsTestSuite(object):
self.assertIn("error", json) self.assertIn("error", json)
def create_and_delete(self): def create_and_delete(self):
app = "create_and_delete"
create_json, create_status = self.testloop.run_until_complete( create_json, create_status = self.testloop.run_until_complete(
self.test_client.apps.create("testapp")) self.test_client.apps.create(app))
delete_json, delete_status = self.testloop.run_until_complete( delete_json, delete_status = self.testloop.run_until_complete(
self.test_client.apps.delete(create_json["app"]["name"])) self.test_client.apps.delete(create_json["app"]["name"]))
@ -42,7 +43,7 @@ class AppsTestSuite(object):
self.assertEqual(200, delete_status) self.assertEqual(200, delete_status)
def attempt_to_double_create(self): def attempt_to_double_create(self):
app = "testapp" app = "attempt_to_double_create"
create_json, _ = self.testloop.run_until_complete( create_json, _ = self.testloop.run_until_complete(
self.test_client.apps.create(app)) self.test_client.apps.create(app))
err, status = self.testloop.run_until_complete( err, status = self.testloop.run_until_complete(
@ -60,7 +61,7 @@ class AppsTestSuite(object):
self.assertIn("error", json) self.assertIn("error", json)
def delete_with_routes(self): def delete_with_routes(self):
app_name = "testapp" app_name = "delete_with_routes"
app, _ = self.testloop.run_until_complete( app, _ = self.testloop.run_until_complete(
self.test_client.apps.create(app_name)) self.test_client.apps.create(app_name))
self.testloop.run_until_complete( self.testloop.run_until_complete(
@ -82,3 +83,16 @@ class AppsTestSuite(object):
self.assertIn("message", attempt["error"]) self.assertIn("message", attempt["error"])
self.assertIn("with routes", attempt["error"]["message"]) self.assertIn("with routes", attempt["error"]["message"])
self.assertEqual(200, status_2) self.assertEqual(200, status_2)
def update_app(self):
app_name = "update_app"
app, _ = self.testloop.run_until_complete(
self.test_client.apps.create(app_name))
_, update_status = self.testloop.run_until_complete(
self.test_client.apps.update(
app["app"]["name"], config={}
)
)
self.testloop.run_until_complete(
self.test_client.apps.delete(app["app"]["name"]))
self.assertEqual(200, update_status)

View File

@ -16,14 +16,14 @@ import asyncio
import datetime import datetime
import uvloop import uvloop
from laos.common import logger as log from ...common import logger as log
class LaosTestsBase(object): class LaosTestsBase(object):
def get_loop_and_logger(self, test_type): def get_loop_and_logger(self, test_type):
self.route_data = { self.route_data = {
"type": "sync", "type": "async",
"path": "/hello-sync-private", "path": "/hello-sync-private",
"image": "iron/hello", "image": "iron/hello",
"is_public": "false" "is_public": "false"

View File

@ -44,6 +44,10 @@ class AppsV1(object):
async def delete(self, app_name): async def delete(self, app_name):
return await self.client.remove(self.app_path, app_name) return await self.client.remove(self.app_path, app_name)
async def update(self, app_name, **data):
return await self.client.update(
self.app_path, app_name, **data)
class RoutesV1(object): class RoutesV1(object):
@ -51,6 +55,10 @@ class RoutesV1(object):
routes_path = "/v1/{}/apps/{}/routes" routes_path = "/v1/{}/apps/{}/routes"
# /v1/{project_id}/apps/{app}/routes{} # /v1/{project_id}/apps/{app}/routes{}
route_path = routes_path + "{}" route_path = routes_path + "{}"
# /v1/r/{project_id}/{app}{route}
private_execution_path = "/v1/r/{}/{}{}"
# /r/{app}{route}
public_execution_path = "/r/{}{}"
def __init__(self, test_client): def __init__(self, test_client):
self.client = test_client self.client = test_client
@ -71,6 +79,19 @@ class RoutesV1(object):
return await self.client.remove( return await self.client.remove(
self.route_path, app_name, path) self.route_path, app_name, path)
async def update(self, app_name, path, **data):
return await self.client.update(
self.route_path, app_name, path, **data)
async def execute_private(self, app_name, path, **data):
return await self.client.execute(
self.private_execution_path, app_name, path, **data)
async def execute_public(self, app_name, path, **data):
return await self.client.execute(
self.public_execution_path, app_name, path,
ignore_project_id=True, **data)
class ProjectBoundLaosTestClient(test_utils.TestClient): class ProjectBoundLaosTestClient(test_utils.TestClient):
@ -113,3 +134,22 @@ class ProjectBoundLaosTestClient(test_utils.TestClient):
headers=self.headers) headers=self.headers)
json = await resp.json() json = await resp.json()
return json, resp.status return json, resp.status
async def update(self, route_path, resource_name, *parts, **data):
resp = await self.put(
route_path.format(self.project_id, resource_name, *parts),
data=jsonlib.dumps(data),
headers=self.headers,)
json = await resp.json()
return json, resp.status
async def execute(self, route, resource_name, *parts,
ignore_project_id=False, **data):
if not ignore_project_id:
route = route.format(self.project_id, resource_name, *parts)
else:
route = route.format(resource_name, *parts)
resp = await self.post(
route, data=jsonlib.dumps(data), headers=self.headers)
json = await resp.json()
return json, resp.status

View File

@ -12,9 +12,40 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
import contextlib
import json as jsonlib import json as jsonlib
@contextlib.contextmanager
def setup_execute(self, app_name):
app, _ = self.testloop.run_until_complete(
self.test_client.apps.create(app_name)
)
new_app_name = app["app"]["name"]
route, _ = self.testloop.run_until_complete(
self.test_client.routes.create(
new_app_name, **self.route_data)
)
self.testloop.run_until_complete(
self.test_client.routes.update(
new_app_name, self.route_data["path"], **{
"type": "sync"
}
)
)
try:
yield new_app_name
except Exception as ex:
print(ex)
finally:
self.testloop.run_until_complete(
self.test_client.routes.delete(
new_app_name, self.route_data["path"])
)
self.testloop.run_until_complete(
self.test_client.apps.delete(new_app_name))
class AppRoutesTestSuite(object): class AppRoutesTestSuite(object):
def list_routes_from_unknown_app(self): def list_routes_from_unknown_app(self):
@ -26,8 +57,9 @@ class AppRoutesTestSuite(object):
self.assertIn("not found", json["error"]["message"]) self.assertIn("not found", json["error"]["message"])
def list_routes_from_existing_app(self): def list_routes_from_existing_app(self):
app = "list_routes_from_existing_app"
create_json, _ = self.testloop.run_until_complete( create_json, _ = self.testloop.run_until_complete(
self.test_client.apps.create("testapp")) self.test_client.apps.create(app))
json, status = self.testloop.run_until_complete( json, status = self.testloop.run_until_complete(
self.test_client.routes.list(create_json["app"]["name"]) self.test_client.routes.list(create_json["app"]["name"])
) )
@ -38,9 +70,10 @@ class AppRoutesTestSuite(object):
self.assertIn("message", json) self.assertIn("message", json)
def show_unknown_route_from_existing_app(self): def show_unknown_route_from_existing_app(self):
app = "show_unknown_route_from_existing_app"
path = "/unknown_path" path = "/unknown_path"
create_json, _ = self.testloop.run_until_complete( create_json, _ = self.testloop.run_until_complete(
self.test_client.apps.create("testapp")) self.test_client.apps.create(app))
json, status = self.testloop.run_until_complete( json, status = self.testloop.run_until_complete(
self.test_client.routes.show( self.test_client.routes.show(
create_json["app"]["name"], path) create_json["app"]["name"], path)
@ -53,8 +86,9 @@ class AppRoutesTestSuite(object):
self.assertIn("not found", json["error"]["message"]) self.assertIn("not found", json["error"]["message"])
def delete_unknown_route_from_existing_app(self): def delete_unknown_route_from_existing_app(self):
app = "delete_unknown_route_from_existing_app"
create_json, _ = self.testloop.run_until_complete( create_json, _ = self.testloop.run_until_complete(
self.test_client.apps.create("testapp")) self.test_client.apps.create(app))
json, status = self.testloop.run_until_complete( json, status = self.testloop.run_until_complete(
self.test_client.routes.delete( self.test_client.routes.delete(
create_json["app"]["name"], "/unknown_path") create_json["app"]["name"], "/unknown_path")
@ -67,18 +101,23 @@ class AppRoutesTestSuite(object):
self.assertIn("not found", json["error"]["message"]) self.assertIn("not found", json["error"]["message"])
def create_and_delete_route(self): def create_and_delete_route(self):
app, _ = self.testloop.run_until_complete( app_name = "create_and_delete_route"
self.test_client.apps.create("testapp")) created, _ = self.testloop.run_until_complete(
self.test_client.apps.create(app_name))
new_app_name = created["app"]["name"]
route, create_status = self.testloop.run_until_complete( route, create_status = self.testloop.run_until_complete(
self.test_client.routes.create( self.test_client.routes.create(
app["app"]["name"], **self.route_data) new_app_name, **self.route_data)
) )
route_deleted, delete_status = self.testloop.run_until_complete( route_deleted, delete_status = self.testloop.run_until_complete(
self.test_client.routes.delete( self.test_client.routes.delete(
app["app"]["name"], self.route_data["path"]) new_app_name, self.route_data["path"])
) )
self.testloop.run_until_complete( self.testloop.run_until_complete(
self.test_client.apps.delete(app["app"]["name"])) self.test_client.apps.delete(new_app_name))
print(route)
after_post = route["route"] after_post = route["route"]
for k in self.route_data: for k in self.route_data:
if k == "path": if k == "path":
@ -94,25 +133,77 @@ class AppRoutesTestSuite(object):
self.assertIn("message", route_deleted) self.assertIn("message", route_deleted)
def double_create_route(self): def double_create_route(self):
app, _ = self.testloop.run_until_complete( app = "double_create_route"
self.test_client.apps.create("testapp")) created_app, _ = self.testloop.run_until_complete(
self.test_client.apps.create(app))
new_app_name = created_app["app"]["name"]
self.testloop.run_until_complete( self.testloop.run_until_complete(
self.test_client.routes.create( self.test_client.routes.create(
app["app"]["name"], **self.route_data) new_app_name, **self.route_data)
) )
json, double_create_status = self.testloop.run_until_complete( json, double_create_status = self.testloop.run_until_complete(
self.test_client.routes.create( self.test_client.routes.create(
app["app"]["name"], **self.route_data) new_app_name, **self.route_data)
) )
self.testloop.run_until_complete( self.testloop.run_until_complete(
self.test_client.routes.delete( self.test_client.routes.delete(
app["app"]["name"], self.route_data["path"]) new_app_name, self.route_data["path"])
) )
self.testloop.run_until_complete( self.testloop.run_until_complete(
self.test_client.apps.delete(app["app"]["name"])) self.test_client.apps.delete(new_app_name))
self.assertEqual(409, double_create_status) self.assertEqual(409, double_create_status)
self.assertIn("error", json) self.assertIn("error", json)
self.assertIn("message", json["error"]) self.assertIn("message", json["error"])
self.assertIn("already exist", json["error"]["message"]) self.assertIn("already exist", json["error"]["message"])
def update_route(self):
app = "update_route"
created, _ = self.testloop.run_until_complete(
self.test_client.apps.create(app)
)
new_app_name = created["app"]["name"]
route, _ = self.testloop.run_until_complete(
self.test_client.routes.create(
new_app_name, **self.route_data)
)
print(route)
updated, update_status = self.testloop.run_until_complete(
self.test_client.routes.update(
new_app_name, self.route_data["path"], **{
"type": "sync"
}
)
)
print(updated)
self.testloop.run_until_complete(
self.test_client.routes.delete(
new_app_name, self.route_data["path"])
)
self.testloop.run_until_complete(
self.test_client.apps.delete(new_app_name))
self.assertEqual(200, update_status)
self.assertNotIn(route["route"]["type"], updated["route"]["type"])
def execute_private(self):
with setup_execute(self, "execute_private") as app_name:
result, status = self.testloop.run_until_complete(
self.test_client.routes.execute_private(
app_name, self.route_data["path"]
)
)
self.assertIsNotNone(result)
self.assertEqual(200, status)
def execute_public(self):
with setup_execute(self, "execute_public") as app_name:
result, status = self.testloop.run_until_complete(
self.test_client.routes.execute_public(
app_name, self.route_data["path"]
)
)
self.assertIsNotNone(result)
self.assertEqual(200, status)

View File

@ -69,18 +69,27 @@ class FakeRoutes(object):
) )
) )
async def execute(self, path, loop=None): async def execute(self, path, loop=None, **data):
app_routes = APP_ROUTES[self.app_name] app_routes = APP_ROUTES[self.app_name]
if path not in [route.path for route in app_routes]: if path not in [route.path for route in app_routes]:
raise client.FunctionsAPIException( raise client.FunctionsAPIException(
"App {} route {} not found.".format( "App {} route {} not found.".format(
self.app_name, path), 404) self.app_name, path), 404)
else: else:
route = await self.show(path) route = await self.show(path, loop=loop)
return "Hello world!" if route.type == "sync" else { return "Hello world!" if route.type == "sync" else {
"call_id": uuid.uuid4().hex "call_id": uuid.uuid4().hex
} }
async def update(self, route_path, loop=None, **data):
route = await self.show(route_path, loop=loop)
if "path" in data:
del data['path']
route.__kwargs__.update(data)
for k, v in route.__kwargs__.items():
setattr(route, k, v)
return route
class FakeApps(object): class FakeApps(object):
@ -104,17 +113,29 @@ class FakeApps(object):
async def show(self, app_name, loop=None): async def show(self, app_name, loop=None):
if app_name not in APPS: if app_name not in APPS:
raise client.FunctionsAPIException( raise client.FunctionsAPIException(
"App {} already exist.".format(app_name), 404) "App {} not found.".format(app_name), 404)
else: else:
return APPS.get(app_name) return APPS.get(app_name)
async def delete(self, app_name, loop=None): async def delete(self, app_name, loop=None):
if app_name not in APPS: if app_name not in APPS:
raise client.FunctionsAPIException( raise client.FunctionsAPIException(
"App {} already exist.".format(app_name), 404) "App {} not found.".format(app_name), 404)
else: else:
if APP_ROUTES[app_name]:
raise client.FunctionsAPIException(
"Cannot remove apps with routes", 403)
del APPS[app_name] del APPS[app_name]
async def update(self, app_name, loop=None, **data):
app = await self.show(app_name, loop=loop)
if 'name' in data:
del data['name']
app.__kwargs__.update(data)
for k, v in app.__kwargs__.items():
setattr(app, k, v)
return app
class FunctionsAPIV1(object): class FunctionsAPIV1(object):

View File

@ -12,24 +12,23 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
import collections
import os import os
import testtools import testtools
import uuid import uuid
from aioservice.http import service from aioservice.http import service
from laos.api.controllers import apps from ...api.controllers import apps
from laos.api.controllers import routes from ...api.controllers import routes
from laos.api.controllers import runnable from ...api.controllers import runnable
from laos.api.controllers import tasks from ...api.controllers import tasks
from laos.api.middleware import content_type from ...api.middleware import content_type
from laos.common import config from ...common import config
from laos.tests.common import base from ..common import base
from laos.tests.common import client from ..common import client
from laos.tests.fakes import functions_api from ..fakes import functions_api
class LaosFunctionalTestsBase(base.LaosTestsBase, testtools.TestCase): class LaosFunctionalTestsBase(base.LaosTestsBase, testtools.TestCase):
@ -78,20 +77,9 @@ class LaosFunctionalTestsBase(base.LaosTestsBase, testtools.TestCase):
self.test_client = client.ProjectBoundLaosTestClient( self.test_client = client.ProjectBoundLaosTestClient(
self.testapp, self.project_id) self.testapp, self.project_id)
self.route_data = {
"type": "sync",
"path": "/hello-sync-private",
"image": "iron/hello",
"is_public": "false"
}
self.testloop.run_until_complete(self.test_client.start_server()) self.testloop.run_until_complete(self.test_client.start_server())
super(LaosFunctionalTestsBase, self).setUp() super(LaosFunctionalTestsBase, self).setUp()
def tearDown(self): def tearDown(self):
functions_api.APPS = {}
functions_api.ROUTES = collections.defaultdict(list)
# ^ temporary solution,
# until https://github.com/iron-io/functions/issues/274 fixed
self.testloop.run_until_complete(self.test_client.close()) self.testloop.run_until_complete(self.test_client.close())
super(LaosFunctionalTestsBase, self).tearDown() super(LaosFunctionalTestsBase, self).tearDown()

View File

@ -12,8 +12,8 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
from laos.tests.common import apps as apps_suite from ..common import apps as apps_suite
from laos.tests.functional import base from ..functional import base
class TestApps(base.LaosFunctionalTestsBase, apps_suite.AppsTestSuite): class TestApps(base.LaosFunctionalTestsBase, apps_suite.AppsTestSuite):
@ -35,3 +35,6 @@ class TestApps(base.LaosFunctionalTestsBase, apps_suite.AppsTestSuite):
def test_delete_with_routes(self): def test_delete_with_routes(self):
super(TestApps, self).delete_with_routes() super(TestApps, self).delete_with_routes()
def test_update_app(self):
super(TestApps, self).update_app()

View File

@ -12,8 +12,8 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
from laos.tests.common import routes as routes_suite from ..common import routes as routes_suite
from laos.tests.functional import base from ..functional import base
class TestAppRoutes(base.LaosFunctionalTestsBase, class TestAppRoutes(base.LaosFunctionalTestsBase,
@ -40,3 +40,12 @@ class TestAppRoutes(base.LaosFunctionalTestsBase,
def test_double_create_route(self): def test_double_create_route(self):
super(TestAppRoutes, self).double_create_route() super(TestAppRoutes, self).double_create_route()
def test_update_route(self):
super(TestAppRoutes, self).update_route()
def test_private_execution(self):
super(TestAppRoutes, self).execute_private()
def test_public_execution(self):
super(TestAppRoutes, self).execute_private()

View File

@ -12,18 +12,19 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
import aiohttp
import json import json
import os import os
from urllib import parse
import aiohttp
import testtools import testtools
from laos.common import config from ...common import config
from laos.service import laos_api from ..common import base
from ..common import client
from laos.tests.common import base
from laos.tests.common import client
from urllib import parse from service import laos_api
class LaosIntegrationTestsBase(base.LaosTestsBase, testtools.TestCase): class LaosIntegrationTestsBase(base.LaosTestsBase, testtools.TestCase):

View File

@ -12,8 +12,8 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
from laos.tests.common import apps as apps_suite from ..common import apps as apps_suite
from laos.tests.integration import base from ..integration import base
class TestIntegrationApps(base.LaosIntegrationTestsBase, class TestIntegrationApps(base.LaosIntegrationTestsBase,
@ -36,3 +36,6 @@ class TestIntegrationApps(base.LaosIntegrationTestsBase,
def test_delete_with_routes(self): def test_delete_with_routes(self):
super(TestIntegrationApps, self).delete_with_routes() super(TestIntegrationApps, self).delete_with_routes()
def test_update_app(self):
super(TestIntegrationApps, self).update_app()

View File

@ -12,8 +12,8 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
from laos.tests.common import routes as routes_suite from ..common import routes as routes_suite
from laos.tests.functional import base from ..functional import base
class TestIntegrationAppRoutes(base.LaosFunctionalTestsBase, class TestIntegrationAppRoutes(base.LaosFunctionalTestsBase,
@ -45,6 +45,13 @@ class TestIntegrationAppRoutes(base.LaosFunctionalTestsBase,
).create_and_delete_route() ).create_and_delete_route()
def test_double_create_route(self): def test_double_create_route(self):
super( super(TestIntegrationAppRoutes, self).double_create_route()
TestIntegrationAppRoutes, self
).double_create_route() def test_update_route(self):
super(TestIntegrationAppRoutes, self).update_route()
def test_private_execution(self):
super(TestIntegrationAppRoutes, self).execute_private()
def test_public_execution(self):
super(TestIntegrationAppRoutes, self).execute_private()

View File

@ -3,13 +3,13 @@
# process, which may cause wedges in the gate later. # process, which may cause wedges in the gate later.
uvloop==0.6.0 # Apache-2.0 uvloop==0.6.0 # Apache-2.0
aioservice==0.0.1 # Apache-2.0 aioservice==0.0.2 # Apache-2.0
aiomysql==0.0.9 # Apache-2.0 aiomysql==0.0.9 # Apache-2.0
alembic==0.8.8 # MIT alembic==0.8.8 # MIT
click==6.6 # Apache-2.0 click==6.6 # Apache-2.0
# IronFunctions # IronFunctions
python-functionsclient==0.0.1 python-functionsclient==0.0.2
# OpenStack # OpenStack
keystoneauth1==2.15.0 # Apache-2.0 keystoneauth1==2.15.0 # Apache-2.0

18
scripts/docker_full.sh Executable file
View File

@ -0,0 +1,18 @@
#!/usr/bin/env bash
set +x
set +e
set +i
docker build -t laos-api -f Dockerfile . 2>&1
docker_h=$(echo ${DOCKER_HOST} | tr "://" " " |awk '{print $2}')
echo -e "Docker IP address ${docker_h}"
echo -e "OpenStack Identity service URL ${OS_AUTH_URL}"
echo -e "IronFunctions URL ${FUNCTIONS_API_URL}"
echo -e "Persistent storage URI ${TEST_DB_URI}"
docker run -d -p ${docker_h}:10002:10001 --env LAOS_HOST=0.0.0.0 --env LAOS_PORT=10001 --env LAOS_DB=${TEST_DB_URI} --env KEYSTONE_ENDPOINT=${OS_AUTH_URL} --env FUNCTIONS_URL=${FUNCTIONS_API_URL} --env LAOS_LOG_LEVEL=INFO laos-api
sleep 2
docker ps
echo -e "Service running on ${docker_h}:10002"
curl -X GET http://${docker_h}:10002/api/swagger.json | python -mjson.tool
docker stop -t 1 $(docker ps | grep "${docker_h}:10002" | awk '{print $1}')

View File

@ -30,11 +30,11 @@ setuptools.setup(
packages=setuptools.find_packages(), packages=setuptools.find_packages(),
install_requires=[ install_requires=[
"uvloop==0.6.0", "uvloop==0.6.0",
"aioservice==0.0.1", "aioservice==0.0.2",
"aiomysql==0.0.9", "aiomysql==0.0.9",
"alembic==0.8.8", "alembic==0.8.8",
"click==6.6", "click==6.6",
"python-functionsclient==0.0.1", "python-functionsclient==0.0.2",
"keystoneauth1==2.15.0", "keystoneauth1==2.15.0",
"python-keystoneclient==3.6.0", "python-keystoneclient==3.6.0",
"aiohttp-swagger==1.0.2", "aiohttp-swagger==1.0.2",
@ -46,8 +46,6 @@ setuptools.setup(
'Intended Audience :: System Administrators', 'Intended Audience :: System Administrators',
'Intended Audience :: Developers', 'Intended Audience :: Developers',
'Environment :: No Input/Output (Daemon)', 'Environment :: No Input/Output (Daemon)',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.5',
'Topic :: Software Development :: ' 'Topic :: Software Development :: '
'Libraries :: Python Modules', 'Libraries :: Python Modules',

107
testing.md Normal file
View File

@ -0,0 +1,107 @@
Testing
-------
In order to run tests you need to install `Tox`:
$ pip install tox
Also, you will need a running MySQL instance with the database migrations applied from the previous step.
Tests are dependent on pre-created MySQL database for persistence.
Please set env var
$ export TEST_DB_URI=mysql://<your-user>:<your-user-password>@<mysql-host>:<mysql-port>/<functions-db>
PEP8 style checks
-----------------
In order to run `PEP8` style checks run following command:
$ tox -e pep8
Functional testing
------------------
In order to run `functional` tests run following command:
$ tox -e py35-functional
Pros:
* lightweight (controllers and DB models testing)
* no OpenStack required
* no IronFunctions required
Cons:
* MySQL server required
* OpenStack authentication is not tested
* IronFunctions API stubbed with fake implementation
Integration integrations
------------------------
Integration tests are dependent on following env variables:
* TEST_DB_URI - similar to functional tests, database endpoint
* FUNCTIONS_API_URL - IronFunctions API URL (default value - `http://localhost:8080/v1`)
* OS_AUTH_URL - OpenStack Identity endpoint
* OS_PROJECT_NAME - OpenStack user-specific project name
* OS_USERNAME - OpenStack user name
* OS_PASSWORD - OpenStack user user password
To run tests use following command:
export TEST_DB_URI=mysql://<your-user>:<your-user-password>@<mysql-host>:<mysql-port>/<functions-db>
export FUNCTIONS_API_URL=<functions-api-protocol>://<functions-host>:<functions-port>/<functions-api-version>
export OS_AUTH_URL=<identity-api-protocol>://<identity-host>:<identity-port>/<identity-api-version>
export OS_PROJECT_NAME=<project-name>
export OS_USERNAME=<project-name>
export OS_PASSWORD=<project-name>
tox -epy35-integration
Testing: Docker-build
---------------------
This type of testing allows to ensure if code can be build inside docker container with no problems.
In order to run this check use following commands::
export DOCKER_HOST=tcp://<docker-host>:<docker-port>>
export TEST_DB_URI=mysql://<your-user>:<your-user-password>@<mysql-host>:<mysql-port>/<functions-db>
export FUNCTIONS_API_URL=<functions-api-protocol>://<functions-host>:<functions-port>/<functions-api-version>
export OS_AUTH_URL=<identity-api-protocol>://<identity-host>:<identity-port>/<identity-api-version>
tox -e docker-build
During this check Tox:
* builds an image
* deletes all artifacts (Python3.5 image and recently built image)
Testing Docker-full
-------------------
This type of testing allows to ensure if code code can be build and run successfully inside docker container with no problems.
In order to run this check use following commands::
export DOCKER_HOST=tcp://<docker-host>:<docker-port>>
export TEST_DB_URI=mysql://<your-user>:<your-user-password>@<mysql-host>:<mysql-port>/<functions-db>
export FUNCTIONS_API_URL=<functions-api-protocol>://<functions-host>:<functions-port>/<functions-api-version>
export OS_AUTH_URL=<identity-api-protocol>://<identity-host>:<identity-port>/<identity-api-version>
tox -e docker-full
During this check following operations are performed::
* build container from source code
* run container with exposed ports
* request Swagger API doc to see if API is responsive
* tear-down running container
Coverage regression testing
---------------------------
In order to build quality software it is necessary to keep test coverage at its highest point.
So, as part of `Tox` testing new check was added - functional test coverage regression.
In order to run it use following command:
$ tox -e py35-functional-regression

11
tox.ini
View File

@ -1,7 +1,7 @@
# Project LaOS # Project LaOS
[tox] [tox]
envlist = py35-functional,py35-functional-regression,py35-integration,py35-integration-regression,pep8 envlist = py35-functional,py35-functional-regression,py35-integration,py35-integration-regression,pep8,docker-build
minversion = 1.6 minversion = 1.6
skipsdist = True skipsdist = True
@ -14,16 +14,16 @@ passenv =
OS_PASSWORD OS_PASSWORD
OS_USERNAME OS_USERNAME
OS_PROJECT_NAME OS_PROJECT_NAME
DOCKER_HOST
setenv = VIRTUAL_ENV={envdir} setenv = VIRTUAL_ENV={envdir}
usedevelop = True usedevelop = True
install_command = pip install -U {opts} {packages} install_command = pip install -U {opts} {packages}
deps = -r{toxinidir}/requirements.txt deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt -r{toxinidir}/test-requirements.txt
commands = find . -type f -name "*.pyc" -delete commands = find . -type f -name "*.pyc" -delete
rm -f .testrepository/times.dbm
python setup.py testr --testr-args='{posargs}'
whitelist_externals = find whitelist_externals = find
rm rm
docker
[testenv:pep8] [testenv:pep8]
commands = flake8 commands = flake8
@ -48,7 +48,10 @@ commands =
rm -rf doc/html doc/build rm -rf doc/html doc/build
python setup.py build_sphinx python setup.py build_sphinx
[testenv:docker-full]
commands = {toxinidir}/scripts/docker_full.sh
[flake8] [flake8]
ignore = H202,H404,H405,H501 ignore = H202,H304,H404,H405,H501
show-source = True show-source = True
exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build,migrations exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build,migrations