From 36d1d1ac682c75167e5fe054f16eefe64988e3cf Mon Sep 17 00:00:00 2001 From: Rob Cresswell Date: Thu, 6 Oct 2016 14:27:22 +0100 Subject: [PATCH] Refactor tox & update docs - Updated tox envlist, so just running `tox` from the CLI will now run all voting gate tests - Reduce duplicated definitions and commands - Remove any reliance on run_tests within tox - Removes all doc references to run_tests.sh, and replaces them with their tox equivalent. Where necessary, language around the tox commands has been altered or extended so that it makes sense and is consistent with other parts of the docs. Also adds a new "Test Environment" list to the docs, so that newcomers do not have to piece together CLI commands and their cryptic extensions from tox.ini - Move the inline shell scripting to its own file. Also fixes a bug when passing args, since the logic assumed you were attempting a subset test run (try `tox -e py27 -- --pdb` on master to compare) - Moved translation tooling from run_tests to manage.py, w/ help text and arg restrictions. This is much more flexible so that plugins can use it without having to copy commands, but still defaults to exactly the same parameters/behaviour from run_tests. Docs updated appropriately. - Removed npm/karma strange reliance on either .venv or tox/py27. Now it only uses tox/npm. Change-Id: I883f885bd424955d39ddcfde5ba396a88cfc041e Implements: blueprint enhance-tox Closes-Bug: 1638672 --- .gitignore | 1 + doc/source/contributing.rst | 7 +- doc/source/quickstart.rst | 28 ++-- doc/source/ref/run_tests.rst | 6 + doc/source/testing.rst | 107 +++++++++++-- doc/source/topics/angularjs.rst | 11 +- doc/source/topics/customizing.rst | 2 +- doc/source/topics/install.rst | 2 +- doc/source/topics/javascript_testing.rst | 9 +- doc/source/topics/translation.rst | 16 +- doc/source/tutorials/dashboard.rst | 24 ++- doc/source/tutorials/table_actions.rst | 4 +- horizon/karma.conf.js | 17 +- openstack_dashboard/karma.conf.js | 17 +- .../management/commands/extract_messages.py | 57 +++++++ .../management/commands/update_catalog.py | 122 +++++++++++++++ package.json | 2 +- .../bp/enhance-tox-26f73a048b88df2f.yaml | 11 ++ tools/pseudo.py | 4 + tools/unit_tests.sh | 30 ++++ tox.ini | 147 +++++++----------- 21 files changed, 437 insertions(+), 187 deletions(-) create mode 100644 openstack_dashboard/management/commands/extract_messages.py create mode 100644 openstack_dashboard/management/commands/update_catalog.py create mode 100644 releasenotes/notes/bp/enhance-tox-26f73a048b88df2f.yaml create mode 100755 tools/unit_tests.sh diff --git a/.gitignore b/.gitignore index 8a1c59f3cc..78303126aa 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ *.egg* *.mo +*.pot *.pyc *.sw? *.sqlite3 diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst index fc6380097d..05c6255053 100644 --- a/doc/source/contributing.rst +++ b/doc/source/contributing.rst @@ -89,9 +89,8 @@ After You Write Your Patch Once you've made your changes, there are a few things to do: -* Make sure the unit tests pass: ``./run_tests.sh`` for Python, and ``npm run test`` for JS. -* Make sure the linting tasks pass: ``./run_tests.sh --pep8`` for Python, and ``npm run lint`` for JS. -* Make sure your code is ready for translation: ``./run_tests.sh --pseudo de`` See :ref:`pseudo_translation` for more information. +* Make sure the unit tests and linting tasks pass by running ``tox`` +* Make sure your code is ready for translation: See :ref:`pseudo_translation`. * Make sure your code is up-to-date with the latest master: ``git pull --rebase`` * Finally, run ``git review`` to upload your changes to Gerrit for review. @@ -132,7 +131,7 @@ Python ------ We follow PEP8_ for all our Python code, and use ``pep8.py`` (available -via the shortcut ``./run_tests.sh --pep8``) to validate that our code +via the shortcut ``tox -e pep8``) to validate that our code meets proper Python style guidelines. .. _PEP8: http://www.python.org/dev/peps/pep-0008/ diff --git a/doc/source/quickstart.rst b/doc/source/quickstart.rst index d5924ba54c..0a5f765b18 100644 --- a/doc/source/quickstart.rst +++ b/doc/source/quickstart.rst @@ -22,20 +22,10 @@ On RPM-based distributions (e.g., Fedora/RHEL/CentOS/Scientific Linux):: Setup ===== -To setup a Horizon development environment simply clone the Horizon git -repository from http://github.com/openstack/horizon and execute the -``run_tests.sh`` script from the root folder (see :doc:`ref/run_tests`):: +To begin setting up a Horizon development environment simply clone the Horizon +git repository from https://git.openstack.org/cgit/openstack/horizon.:: - > git clone https://github.com/openstack/horizon.git - > cd horizon - > ./run_tests.sh - -.. note:: - - Running ``run_tests.sh`` will build a virtualenv, ``.venv``, where all the - python dependencies for Horizon are installed and referenced. After the - dependencies are installed, the unit test suites in the Horizon repo will be - executed. There should be no errors from the tests. + > git clone https://git.openstack.org/openstack/horizon Next you will need to setup your Django application config by copying ``openstack_dashboard/local/local_settings.py.example`` to ``openstack_dashboard/local/local_settings.py``. To do this quickly you can use the following command:: @@ -92,21 +82,21 @@ order to prevent Conflicts for future migrations:: > mv openstack_dashboard/local/local_settings.diff openstack_dashboard/local/local_settings.diff.old > python manage.py migrate_settings --gendiff -To start the Horizon development server use ``run_tests.sh``:: +To start the Horizon development server use ``tox``:: - > ./run_tests.sh --runserver localhost:9000 + > tox -e runserver .. note:: The default port for runserver is 8000 which is already consumed by - heat-api-cfn in DevStack. If not running in DevStack - `./run_tests.sh --runserver` will start the test server at - `http://localhost:8000`. + heat-api-cfn in DevStack. If running in DevStack + `tox -e runserver -- localhost:9000` will start the test server at + `http://localhost:9000`. .. note:: - The ``run_tests.sh`` script provides wrappers around ``manage.py``. + The ``tox`` environments provide wrappers around ``manage.py``. For more information on manage.py which is a django, see `https://docs.djangoproject.com/en/dev/ref/django-admin/` diff --git a/doc/source/ref/run_tests.rst b/doc/source/ref/run_tests.rst index 951c82e812..7f88e0d8e8 100644 --- a/doc/source/ref/run_tests.rst +++ b/doc/source/ref/run_tests.rst @@ -2,6 +2,12 @@ The ``run_tests.sh`` Script =========================== +.. warning:: + + This script is deprecated as of Newton (11.0), and will be removed in + Queens (13.0), in favor of tox. The tox docs can be found at + https://tox.readthedocs.io/en/latest/ + .. contents:: Contents: :local: diff --git a/doc/source/testing.rst b/doc/source/testing.rst index 78ddf10c0a..d56a3c1a7f 100644 --- a/doc/source/testing.rst +++ b/doc/source/testing.rst @@ -10,34 +10,33 @@ Because Horizon is composed of both the ``horizon`` app and the tests. While they can be run individually without problem, there is an easier way: -Included at the root of the repository is the ``run_tests.sh`` script +Included at the root of the repository is the ``tox.ini`` config which invokes both sets of tests, and optionally generates analyses on both -components in the process. This script is what Jenkins uses to verify the +components in the process. ``tox`` is what Jenkins uses to verify the stability of the project, so you should make sure you run it and it passes before you submit any pull requests/patches. -To run the tests:: +To run all tests:: - $ ./run_tests.sh - -It's also possible to :doc:`run a subset of unit tests`. - -.. seealso:: - - :doc:`ref/run_tests` - Full reference for the ``run_tests.sh`` script. + $ tox +It's also possible to run a subset of the tests. Open ``tox.ini`` in the +Horizon root directory to see a list of test environments. You can read more +about tox in general at https://tox.readthedocs.io/en/latest/. By default running the Selenium tests will open your Firefox browser (you have to install it first, else an error is raised), and you will be able to see the -tests actions. +tests actions:: + + $ tox -e selenium-headless + If you want to run the suite headless, without being able to see them (as they are ran on Jenkins), you can run the tests:: - $ ./run_tests.sh --with-selenium --selenium-headless + $ tox -e selenium-headless Selenium will use a virtual display in this case, instead of your own. In order -to run the tests this way you have to install the dependency `xvfb`, like +to run the tests this way you have to install the dependency `xvfb`, like this:: $ sudo apt-get install xvfb @@ -49,12 +48,90 @@ for a Debian OS flavour, or for Fedora/Red Hat flavours:: If you can't run a virtual display, or would prefer not to, you can use the PhantomJS web driver instead:: - $ ./run_tests.sh --with-selenium --selenium-phantomjs + $ tox -e selenium-phantomjs If you need to install PhantomJS, you may do so with `npm` like this:: $ npm -g install phantomjs +Alternatively, many distributions have system packages for phantomjs, or +it can be downloaded from http://phantomjs.org/download.html. + +tox Test Environments +===================== + +This is a list of test environments available to be executed by +``tox -e ``. + +pep8 +---- + +Runs pep8, which is a tool that checks Python code style. You can read more +about pep8 at https://www.python.org/dev/peps/pep-0008/ + +py27dj18, py27dj19, py27dj110 +----------------------------- + +Runs the Python unit tests against Django 1.8, Django 1.9 and Django 1.10 +respectively + +All other dependencies are as defined by the upper-constraints file at +https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt + +You can run a subset of the tests by passing the test path as an argument to +tox:: + + $ tox -e py27dj18 -- openstack_dashboard.dashboards.identity.users.tests + +You can also pass other arguments. For example, to drop into a live debugger +when a test fails you can use:: + + $ tox -e py27dj18 -- --pdb + +py34 +---- + +Runs the Python unit tests with a Python 3.4 environment. + +py35 +---- + +Runs the Python unit tests with a Python 3.5 environment. + +releasenotes +------------ + +Outputs Horizons release notes as HTML to ``releasenotes/build/html``. + +Also takes an alternative builder as an optional argument, such as +``tox -e docs -- ``, which will output to +``releasenotes/build/``. Available builders are listed at +http://www.sphinx-doc.org/en/latest/builders.html + +npm +--- + +Installs the npm dependencies listed in ``package.json`` and runs the +JavaScript tests. Can also take optional arguments, which will be executed +as an npm script following the dependency install, instead of ``test``. + +Example:: + + $ tox -e npm -- lintq + +docs +---- + +Outputs Horizons documentation as HTML to ``doc/build/html``. + +Also takes an alternative builder as an optional argument, such as +``tox -e docs -- ``, which will output to ``doc/build/``. +Available builders are listed at +http://www.sphinx-doc.org/en/latest/builders.html + +Example:: + + $ tox -e docs -- latexpdf Writing tests ============= diff --git a/doc/source/topics/angularjs.rst b/doc/source/topics/angularjs.rst index cadc8e02c1..939370449c 100644 --- a/doc/source/topics/angularjs.rst +++ b/doc/source/topics/angularjs.rst @@ -38,7 +38,7 @@ ESLint ESLint is a tool for identifying and reporting on patterns in your JS code, and is part of the automated tests run by Jenkins. You can run ESLint from the -horizon root directory with ``npm run lint``, or alternatively on a specific +horizon root directory with ``tox -e npm -- lint``, or alternatively on a specific directory or file with ``eslint file.js``. Horizon includes a `.eslintrc` in its root directory, that is used by the @@ -217,10 +217,13 @@ Testing ======= 1. Open /jasmine in a browser. The development server can be run - with``./run_tests.sh --runserver`` from the horizon root directory. -2. ``npm run test`` from the horizon root directory. + with ``tox -e runserver`` from the horizon root directory; by default, this will + run the development server at ``http://localhost:8000``. +2. ``tox -e npm`` from the horizon root directory. -The code linting job can be run with ``npm run lint``. +The code linting job can be run with ``tox -e npm -- lint``. If there are many +warnings, you can also use ``tox -e npm -- lintq`` to see only errors and +ignore warnings. For more detailed information, see :doc:`javascript_testing`. diff --git a/doc/source/topics/customizing.rst b/doc/source/topics/customizing.rst index 4c5d0ef618..9115b0e7ed 100644 --- a/doc/source/topics/customizing.rst +++ b/doc/source/topics/customizing.rst @@ -68,7 +68,7 @@ theme's ``_variables.scss``:: @import "/themes/default/variables"; Once you have made your changes you must re-generate the static files with - ``./run_tests.sh -m collectstatic``. + ``tox -e manage -- collectstatic``. By default, all of the themes configured by ``AVAILABLE_THEMES`` setting are collected by horizon during the `collectstatic` process. By default, the themes diff --git a/doc/source/topics/install.rst b/doc/source/topics/install.rst index 129791943b..91104afe65 100644 --- a/doc/source/topics/install.rst +++ b/doc/source/topics/install.rst @@ -39,7 +39,7 @@ Installation message catalogs:: $ sudo apt-get install gettext - $ ./run_tests.sh --compilemessages + $ tox -e manage -- compilemessages This command compiles translation message catalogs within Python virtualenv named ``.venv``. After this step, you can remove diff --git a/doc/source/topics/javascript_testing.rst b/doc/source/topics/javascript_testing.rst index d0d3d0855e..1fd702d34a 100644 --- a/doc/source/topics/javascript_testing.rst +++ b/doc/source/topics/javascript_testing.rst @@ -31,12 +31,13 @@ Running Tests Tests can be run in two ways: 1. Open /jasmine in a browser. The development server can be - run with ``./run_tests.sh --runserver`` from the horizon root directory. - 2. ``npm run test`` from the horizon root directory. This runs Karma, + run with ``tox -e runserver`` from the horizon root directory. + 2. ``tox -e npm`` from the horizon root directory. This runs Karma, so it will run all the tests against PhantomJS and generate coverage reports. -The code linting job can be run with ``npm run lint``. +The code linting job can be run with ``tox -e npm -- lint``, or +``tox -e npm -- lintq`` to show errors, but not warnings. Coverage Reports ---------------- @@ -45,7 +46,7 @@ Our Karma setup includes a plugin to generate test coverage reports. When developing, be sure to check the coverage reports on the master branch and compare your development branch; this will help identify missing tests. -To generate coverage reports, run ``npm run test``. The coverage reports can be +To generate coverage reports, run ``tox -e npm``. The coverage reports can be found at ``horizon/coverage-karma/`` (framework tests) and ``openstack_dashboard/coverage-karma/`` (dashboard tests). Load ``/index.html`` in a browser to view the reports. diff --git a/doc/source/topics/translation.rst b/doc/source/topics/translation.rst index 82da561ee3..a3207d0946 100644 --- a/doc/source/topics/translation.rst +++ b/doc/source/topics/translation.rst @@ -46,13 +46,19 @@ translated. Lets break this up into steps we can follow: to locate them. Refer to the guide below on how to use translation and what these markers look like. -2. Once marked, we can then run ``./run_tests.sh --makemessages``, which +2. Once marked, we can then run ``tox -e manage -- extract_messages``, which searches the codebase for these markers and extracts them into a Portable Object Template (POT) file. In horizon, we extract from both the ``horizon`` folder and the ``openstack_dashboard`` folder. We use the AngularJS extractor for JavaScript and HTML files and the Django extractor for Python and Django templates; both extractors are Babel plugins. +3. To update the .po files, you can run ``tox -e manage -- update_catalog`` to + update the .po file for every language, or you can specify a specific + language to update like this: ``tox -e manage -- update_catalog de``. This + is useful if you want to add a few extra translatabale strings for a + downstream customisation. + .. Note :: When pushing code upstream, the only requirement is to mark the strings @@ -242,12 +248,12 @@ translations to validate that your code is ready for translation. Running the pseudo translation tool ----------------------------------- -#. Make sure your English po file is up to date: - ``./run_tests.sh --makemessages`` +#. Make sure your .pot files are up to date: + ``tox -e manage -- extract_messages`` #. Run the pseudo tool to create pseudo translations. For example, to replace the German translation with a pseudo translation: - ``./run_tests.sh --pseudo de`` -#. Compile the catalog: ``./run_tests.sh --compilemessages`` + ``tox -e manage -- update_catalog de --pseudo`` +#. Compile the catalog: ``tox -e manage -- compilemessages`` #. Run your development server. #. Log in and change to the language you pseudo translated. diff --git a/doc/source/tutorials/dashboard.rst b/doc/source/tutorials/dashboard.rst index 96071ae5f4..412f85de76 100644 --- a/doc/source/tutorials/dashboard.rst +++ b/doc/source/tutorials/dashboard.rst @@ -30,20 +30,19 @@ The quick version ----------------- Horizon provides a custom management command to create a typical base -dashboard structure for you. Run the following commands at the same location -where the ``run_tests.sh`` file resides. It generates most of the boilerplate -code you need:: +dashboard structure for you. Run the following commands in your Horizon root +directory. It generates most of the boilerplate code you need:: - mkdir openstack_dashboard/dashboards/mydashboard + $ mkdir openstack_dashboard/dashboards/mydashboard - ./run_tests.sh -m startdash mydashboard \ - --target openstack_dashboard/dashboards/mydashboard + $ tox -e manage -- startdash mydashboard \ + --target openstack_dashboard/dashboards/mydashboard - mkdir openstack_dashboard/dashboards/mydashboard/mypanel + $ mkdir openstack_dashboard/dashboards/mydashboard/mypanel - ./run_tests.sh -m startpanel mypanel \ - --dashboard=openstack_dashboard.dashboards.mydashboard \ - --target=openstack_dashboard/dashboards/mydashboard/mypanel + $ tox -e manage -- startpanel mypanel \ + --dashboard=openstack_dashboard.dashboards.mydashboard \ + --target=openstack_dashboard/dashboards/mydashboard/mypanel You will notice that the directory ``mydashboard`` gets automatically @@ -562,10 +561,9 @@ Run and check the dashboard Everything is in place, now run ``Horizon`` on the different port:: - ./run_tests.sh --runserver 0.0.0.0:8877 + $ tox -e runserver -- 0:9000 - -Go to ``http://:8877`` using a browser. After login as an admin +Go to ``http://:9000`` using a browser. After login as an admin you should be able see ``My Dashboard`` shows up at the left side on horizon. Click it, ``My Group`` will expand with ``My Panel``. Click on ``My Panel``, the right side panel will display an ``Instances Tab`` which has an diff --git a/doc/source/tutorials/table_actions.rst b/doc/source/tutorials/table_actions.rst index 9cbb69a64d..b02d74bd57 100644 --- a/doc/source/tutorials/table_actions.rst +++ b/doc/source/tutorials/table_actions.rst @@ -274,10 +274,10 @@ Run and check the dashboard We must once again run horizon to verify our dashboard is working:: - ./run_tests.sh --runserver 0.0.0.0:8877 + $ tox -e runserver -- 0:9000 -Go to ``http://:8877`` using a browser. After login as an admin, +Go to ``http://:9000`` using a browser. After login as an admin, display ``My Panel`` to see the ``Instances`` table. For every ``ACTIVE`` instance in the table, there will be a ``Create Snapshot`` action on the row. Click on ``Create Snapshot``, enter a snapshot name in the form that is shown, diff --git a/horizon/karma.conf.js b/horizon/karma.conf.js index b8eeed9bc0..e3cc7d3baa 100644 --- a/horizon/karma.conf.js +++ b/horizon/karma.conf.js @@ -20,23 +20,14 @@ var fs = require('fs'); var path = require('path'); module.exports = function (config) { - var xstaticPath; - var basePaths = [ - './.venv', - './.tox/py27' - ]; + var xstaticPath = path.resolve('./.tox/npm'); - for (var i = 0; i < basePaths.length; i++) { - var basePath = path.resolve(basePaths[i]); - - if (fs.existsSync(basePath)) { - xstaticPath = basePath + '/lib/python2.7/site-packages/xstatic/pkg/'; - break; - } + if (fs.existsSync(xstaticPath)) { + xstaticPath += '/lib/python2.7/site-packages/xstatic/pkg/'; } if (!xstaticPath) { - console.error('xStatic libraries not found, please set up venv'); + console.error('xStatic libraries not found, please run `tox -e npm`'); process.exit(1); } diff --git a/openstack_dashboard/karma.conf.js b/openstack_dashboard/karma.conf.js index 0b91931f0a..e0acf2c52e 100644 --- a/openstack_dashboard/karma.conf.js +++ b/openstack_dashboard/karma.conf.js @@ -20,23 +20,14 @@ var fs = require('fs'); var path = require('path'); module.exports = function (config) { - var xstaticPath; - var basePaths = [ - './.venv', - './.tox/py27' - ]; + var xstaticPath = path.resolve('./.tox/npm'); - for (var i = 0; i < basePaths.length; i++) { - var basePath = path.resolve(basePaths[i]); - - if (fs.existsSync(basePath)) { - xstaticPath = basePath + '/lib/python2.7/site-packages/xstatic/pkg/'; - break; - } + if (fs.existsSync(xstaticPath)) { + xstaticPath += '/lib/python2.7/site-packages/xstatic/pkg/'; } if (!xstaticPath) { - console.error('xStatic libraries not found, please set up venv'); + console.error('xStatic libraries not found, please run `tox -e npm`'); process.exit(1); } diff --git a/openstack_dashboard/management/commands/extract_messages.py b/openstack_dashboard/management/commands/extract_messages.py new file mode 100644 index 0000000000..f015f26b38 --- /dev/null +++ b/openstack_dashboard/management/commands/extract_messages.py @@ -0,0 +1,57 @@ +# Copyright 2016 Cisco Systems, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from distutils.dist import Distribution +import os +from subprocess import call + +from django.core.management.base import BaseCommand + + +class Command(BaseCommand): + help = ('Extract strings that have been marked for translation into .POT ' + 'files.') + + def add_arguments(self, parser): + parser.add_argument('-m', '--module', type=str, nargs='+', + default=['openstack_dashboard', 'horizon'], + help=("The target python module(s) to extract " + "strings from")) + parser.add_argument('-d', '--domain', choices=['django', 'djangojs'], + nargs='+', default=['django', 'djangojs'], + help="Domain(s) of the .pot file") + parser.add_argument('--check-only', action='store_true', + help=("Checks that extraction works correctly, " + "then deletes the .pot file to avoid " + "polluting the source code")) + + def handle(self, *args, **options): + cmd = ('python setup.py extract_messages -F babel-{domain}.cfg ' + '-o {module}/locale/{domain}.pot') + distribution = Distribution() + distribution.parse_config_files(distribution.find_config_files()) + + if options['check_only']: + cmd += " ; rm {module}/locale/{domain}.pot" + + for module in options['module']: + for domain in options['domain']: + potfile = '{module}/locale/{domain}.pot'.format(module=module, + domain=domain) + if not os.path.exists(potfile): + with open(potfile, 'wb') as f: + f.write(b'') + + call(cmd.format(module=module, domain=domain, potfile=potfile), + shell=True) diff --git a/openstack_dashboard/management/commands/update_catalog.py b/openstack_dashboard/management/commands/update_catalog.py new file mode 100644 index 0000000000..522dd7f28d --- /dev/null +++ b/openstack_dashboard/management/commands/update_catalog.py @@ -0,0 +1,122 @@ +# coding: utf-8 + +# Copyright 2016 Cisco Systems, Inc. +# Copyright 2015 IBM Corp. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import babel.messages.catalog as catalog +import os +from subprocess import call + +from django.conf import settings +from django.core.management.base import BaseCommand +from django.utils import translation + +LANGUAGE_CODES = [language[0] for language in settings.LANGUAGES + if language[0] != 'en'] +POTFILE = "{module}/locale/{domain}.pot" +POFILE = "{module}/locale/{locale}/LC_MESSAGES/{domain}.po" + + +def translate(segment): + prefix = u"" + # When the id starts with a newline the mo compiler enforces that + # the translated message must also start with a newline. Make + # sure that doesn't get broken when prepending the bracket. + if segment.startswith('\n'): + prefix = u"\n" + orig_size = len(segment) + # Add extra expansion space based on recommendation from + # http://www-01.ibm.com/software/globalization/guidelines/a3.html + if orig_size < 20: + multiplier = 1 + elif orig_size < 30: + multiplier = 0.8 + elif orig_size < 50: + multiplier = 0.6 + elif orig_size < 70: + multiplier = 0.4 + else: + multiplier = 0.3 + extra_length = int(max(0, (orig_size * multiplier) - 10)) + extra_chars = "~" * extra_length + return u"{0}[~{1}~您好яшçあ{2}]".format(prefix, segment, extra_chars) + + +class Command(BaseCommand): + help = 'Update a translation catalog for a specified language' + + def add_arguments(self, parser): + parser.add_argument('-l', '--language', choices=LANGUAGE_CODES, + default=LANGUAGE_CODES, nargs='+', + help=("The language code(s) to pseudo translate")) + parser.add_argument('-m', '--module', type=str, nargs='+', + default=['openstack_dashboard', 'horizon'], + help=("The target python module(s) to extract " + "strings from")) + parser.add_argument('-d', '--domain', choices=['django', 'djangojs'], + nargs='+', default=['django', 'djangojs'], + help="Domain(s) of the .POT file") + parser.add_argument('--pseudo', action='store_true', + help=("Creates a pseudo translation for the " + "specified locale, to check for " + "translatable string coverage")) + + def handle(self, *args, **options): + for module in options['module']: + for domain in options['domain']: + potfile = POTFILE.format(module=module, domain=domain) + + for language in options['language']: + # Get the locale code for the language code given and + # work around broken django conversion function + locales = {'ko': 'ko_KR', 'pl': 'pl_PL', 'tr': 'tr_TR'} + locale = locales.get(language, + translation.to_locale(language)) + pofile = POFILE.format(module=module, locale=locale, + domain=domain) + + # If this isn't a pseudo translation, execute pybabel + if not options['pseudo']: + if not os.path.exists(pofile): + with open(pofile, 'wb') as fobj: + fobj.write(b'') + + cmd = ('pybabel update -l {locale} -i {potfile} ' + '-o {pofile}').format(locale=locale, + potfile=potfile, + pofile=pofile) + call(cmd, shell=True) + continue + + # Pseudo translation logic + with open(potfile, 'r') as f: + pot_cat = pofile.read_po(f, ignore_obsolete=True) + + new_cat = catalog.Catalog(locale=locale, + last_translator="pseudo.py", + charset="utf-8") + num_plurals = new_cat.num_plurals + + for msg in pot_cat: + if msg.pluralizable: + msg.string = [ + translate(u"{}:{}".format(i, msg.id[0])) + for i in range(num_plurals)] + else: + msg.string = translate(msg.id) + new_cat[msg.id] = msg + + with open(pofile, 'w') as f: + pofile.write_po(f, new_cat, ignore_obsolete=True) diff --git a/package.json b/package.json index 495f8766c5..135295ba86 100644 --- a/package.json +++ b/package.json @@ -19,7 +19,7 @@ "karma-threshold-reporter": "0.1.15" }, "scripts": { - "postinstall": "if [ ! -d .venv ]; then tox -epy27 --notest; fi", + "postinstall": "if [ ! -d .tox/npm ]; then tox -e npm --notest; fi", "test": "karma start horizon/karma.conf.js --single-run && karma start openstack_dashboard/karma.conf.js --single-run", "lint": "eslint --no-color openstack_dashboard/static horizon/static openstack_dashboard/dashboards/*/static", "lintq": "eslint --quiet openstack_dashboard/static horizon/static openstack_dashboard/dashboards/*/static" diff --git a/releasenotes/notes/bp/enhance-tox-26f73a048b88df2f.yaml b/releasenotes/notes/bp/enhance-tox-26f73a048b88df2f.yaml new file mode 100644 index 0000000000..9d9f0eff0b --- /dev/null +++ b/releasenotes/notes/bp/enhance-tox-26f73a048b88df2f.yaml @@ -0,0 +1,11 @@ +--- +features: + - The hard-coded run_tests commands for extracting translatable strings and + updating message catalogs have been ported to django management commands + as extract_messages and update_catalog. These accept several parameters + to make them easier to use with downstream customisations and string + modifications, but the default behaviour is the same as before. +deprecations: + - The run_tests.sh script is now deprecated and all functionality has + been provided by either tox or manage.py. run_tests will be removed + in Queens (13.0). diff --git a/tools/pseudo.py b/tools/pseudo.py index fbb14cb76c..4a5ef1a0df 100755 --- a/tools/pseudo.py +++ b/tools/pseudo.py @@ -19,6 +19,10 @@ import argparse import babel.messages.catalog as catalog import babel.messages.pofile as pofile +# NOTE: This implementation has been superseded by the pseudo_translate +# management command, and will be removed in Queens (13.0) when run_tests.sh +# is also removed. + def translate(segment): prefix = u"" diff --git a/tools/unit_tests.sh b/tools/unit_tests.sh new file mode 100755 index 0000000000..edff8acfcb --- /dev/null +++ b/tools/unit_tests.sh @@ -0,0 +1,30 @@ +# Uses envpython and toxinidir from tox run to construct a test command +testcommand="${1} ${2}/manage.py test" +posargs="${@:3}" + +# Attempt to identify if any of the arguments passed from tox is a test subset +if [ -n "$posargs" ]; then + for arg in "$posargs" + do + if [ ${arg:0:1} != "-" ]; then + subset=$arg + fi + done +fi + +# If we are running a test subset, supply the correct settings file. +# If not, simply run the entire test suite. +if [ -n "$subset" ]; then + project="${subset%%.*}" + + if [ $project == "horizon" ]; then + $testcommand --settings=horizon.test.settings $posargs + elif [ $project == "openstack_dashboard" ]; then + $testcommand --settings=openstack_dashboard.test.settings \ + --exclude-dir=openstack_dashboard/test/integration_tests $posargs + fi +else + $testcommand horizon --settings=horizon.test.settings $posargs + $testcommand openstack_dashboard --settings=openstack_dashboard.test.settings \ + --exclude-dir=openstack_dashboard/test/integration_tests $posargs +fi diff --git a/tox.ini b/tox.ini index 72405b1106..3b30f3cd5f 100644 --- a/tox.ini +++ b/tox.ini @@ -1,105 +1,64 @@ [tox] -envlist = pep8,py27dj18,py27,py34,py35,releasenotes -minversion = 1.6 +envlist = pep8,py27dj{18,19,110},py34,py35,releasenotes,npm +minversion = 2.3.2 skipsdist = True [testenv] -basepython=python2.7 install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages} setenv = VIRTUAL_ENV={envdir} INTEGRATION_TESTS=0 - SELENIUM_HEADLESS=0 - SELENIUM_PHANTOMJS=0 NOSE_WITH_OPENSTACK=1 NOSE_OPENSTACK_SHOW_ELAPSED=1 +whitelist_externals = + bash deps = -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -whitelist_externals = - bash commands = - # Try to detect whether a limited test suite is being specified and if so - # direct the testing to that suite's project; otherwise run the full suite - # in both horizon and openstack_dashboard "projects". - bash -c 'project=`echo {posargs} | cut -d. -f1`; \ - if [ -z "$project" ]; then \ - EXIT_STATUS=0; \ - {envpython} {toxinidir}/manage.py test horizon --settings=horizon.test.settings {posargs} || EXIT_STATUS=$?; \ - {envpython} {toxinidir}/manage.py test openstack_dashboard --settings=openstack_dashboard.test.settings --exclude-dir=openstack_dashboard/test/integration_tests {posargs} || EXIT_STATUS=$?; \ - exit $EXIT_STATUS; \ - else \ - {envpython} {toxinidir}/manage.py test {posargs} --settings=$project.test.settings --exclude-dir=openstack_dashboard/test/integration_tests; \ - fi' - -# Django-1.8 is LTS -[testenv:py27dj18] -commands = - pip install django>=1.8,<1.9 - {envpython} {toxinidir}/manage.py test horizon --settings=horizon.test.settings {posargs} - {envpython} {toxinidir}/manage.py test openstack_dashboard --settings=openstack_dashboard.test.settings --exclude-dir=openstack_dashboard/test/integration_tests {posargs} + docs: sphinx-build -W -b html doc/source doc/build/html + horizon: {envpython} {toxinidir}/manage.py test --settings=horizon.test.settings {posargs} + manage: {envpython} {toxinidir}/manage.py {posargs} + py27: {[unit_tests]commands} + py27dj18: {[unit_tests]commands} + py34: {[unit_tests]commands} + py35: {[unit_tests]commands} + openstack_dashboard: {envpython} {toxinidir}/manage.py test --settings=openstack_dashboard.test.settings {posargs} + releasenotes: sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html + runserver: {envpython} {toxinidir}/manage.py runserver {posargs} + venv: {posargs} [testenv:py34] -basepython = python3.4 setenv = + PYTHONUNBUFFERED = 1 {[testenv]setenv} - PYTHONUNBUFFERED=1 +commands = {[unit_tests]commands} [testenv:py35] -basepython = python3.5 setenv = + PYTHONUNBUFFERED = 1 {[testenv]setenv} - PYTHONUNBUFFERED=1 +commands = {[unit_tests]commands} + +[testenv:py27dj19] +commands = + pip install -U django>=1.9,<1.10 + {[unit_tests]commands} + +[testenv:py27dj110] +commands = + pip install -U django>=1.10,<1.11 + {[unit_tests]commands} + +[unit_tests] +commands = bash {toxinidir}/tools/unit_tests.sh {envpython} {toxinidir} {posargs} [testenv:pep8] usedevelop = True -whitelist_externals = - git - rm -setenv = - {[testenv]setenv} - DJANGO_SETTINGS_MODULE=openstack_dashboard.test.settings commands = - {[testenv:extractmessages_check]commands} - {[testenv:docs]commands} + {envpython} {toxinidir}/manage.py extract_messages --check-only flake8 -[testenv:extractmessages] -usedevelop = True -setenv = - {[testenv]setenv} -commands = - {envpython} {toxinidir}/setup.py extract_messages -F babel-django.cfg -o horizon/locale/django.pot --input-dirs horizon/ - {envpython} {toxinidir}/setup.py extract_messages -F babel-djangojs.cfg -o horizon/locale/djangojs.pot --input-dirs horizon/ - {envpython} {toxinidir}/setup.py extract_messages -F babel-django.cfg -o openstack_dashboard/locale/django.pot --input-dirs openstack_dashboard/ - {envpython} {toxinidir}/setup.py extract_messages -F babel-djangojs.cfg -o openstack_dashboard/locale/djangojs.pot --input-dirs openstack_dashboard/ - -[testenv:extractmessages_check] -# Only checks to see if translation files can be extracted and cleans afterwards -usedevelop = True -whitelist_externals = - rm -setenv = - {[testenv]setenv} -commands = - {[testenv:extractmessages]commands} - rm horizon/locale/django.pot - rm horizon/locale/djangojs.pot - rm openstack_dashboard/locale/django.pot - rm openstack_dashboard/locale/djangojs.pot - -[testenv:compilemessages] -usedevelop = False -commands = - /bin/bash {toxinidir}/run_tests.sh --compilemessages -N - -[testenv:venv] -commands = {posargs} - -[testenv:manage] -# Env to launch manage.py commands -commands = {envpython} {toxinidir}/manage.py {posargs} - [testenv:cover] commands = coverage erase @@ -108,13 +67,28 @@ commands = coverage xml coverage html -[testenv:py27dj19] -commands = pip install django>=1.9,<1.10 - /bin/bash run_tests.sh -N --no-pep8 {posargs} +[testenv:selenium] +setenv = + {[testenv]setenv} + WITH_SELENIUM=1 + SKIP_UNITTESTS=1 +commands = {[unit_tests]commands} -[testenv:py27dj110] -commands = pip install django>=1.10,<1.11 - /bin/bash run_tests.sh -N --no-pep8 {posargs} +[testenv:selenium-headless] +setenv = + {[testenv]setenv} + SELENIUM_HEADLESS=1 + WITH_SELENIUM=1 + SKIP_UNITTESTS=1 +commands = {[unit_tests]commands} + +[testenv:selenium-phantomjs] +setenv = + {[testenv]setenv} + SELENIUM_PHANTOMJS=1 + WITH_SELENIUM=1 + SKIP_UNITTESTS=1 +commands = {[unit_tests]commands} [testenv:py27integration] # Run integration tests only @@ -134,16 +108,6 @@ commands = npm install npm run {posargs:test} -[testenv:docs] -setenv = DJANGO_SETTINGS_MODULE=openstack_dashboard.test.settings -commands = sphinx-build -W -b html doc/source doc/build/html - -[testenv:releasenotes] -commands = sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html - -[testenv:runserver] -commands = {envpython} {toxinidir}/manage.py runserver {posargs} - [testenv:tests_system_packages] # Provide an environment for system packagers that dont want anything from pip # Any extra deps needed for this env can be passed by setting TOX_EXTRA_DEPS @@ -153,8 +117,7 @@ passenv = TOX_EXTRA_DEPS deps = commands = pip install -U {env:TOX_EXTRA_DEPS:} - {envpython} {toxinidir}/manage.py test horizon --settings=horizon.test.settings {posargs} - {envpython} {toxinidir}/manage.py test openstack_dashboard --settings=openstack_dashboard.test.settings {posargs} + {[unit_tests]commands} [flake8] exclude = .venv,.git,.tox,dist,*lib/python*,*egg,build,panel_template,dash_template,local_settings.py,*/local/*,*/test/test_plugins/*,.ropeproject,node_modules