commit 2c9c5f005bbd42faf21a317c6336ad142a859bb0 Author: Kevin Carter Date: Tue Dec 8 11:00:15 2015 -0600 first commit Signed-off-by: Kevin Carter diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst new file mode 100644 index 0000000..390643e --- /dev/null +++ b/CONTRIBUTING.rst @@ -0,0 +1,85 @@ +plugins +######## +:tags: openstack, cloud, ansible +:category: \*nix + +contributor guidelines +^^^^^^^^^^^^^^^^^^^^^^ + +Filing Bugs +----------- + +Bugs should be filed on Launchpad, not GitHub: "https://bugs.launchpad.net/openstack-ansible" + + +When submitting a bug, or working on a bug, please ensure the following criteria are met: + * The description clearly states or describes the original problem or root cause of the problem. + * Include historical information on how the problem was identified. + * Any relevant logs are included. + * The provided information should be totally self-contained. External access to web services/sites should not be needed. + * Steps to reproduce the problem if possible. + + +Submitting Code +--------------- + +Changes to the project should be submitted for review via the Gerrit tool, following +the workflow documented at: "http://docs.openstack.org/infra/manual/developers.html#development-workflow" + +Pull requests submitted through GitHub will be ignored and closed without regard. + + +Extra +----- + +Tags: + If it's a bug that needs fixing in a branch in addition to Master, add a '\-backport-potential' tag (eg ``juno-backport-potential``). There are predefined tags that will autocomplete. + +Status: + Please leave this alone, it should be New till someone triages the issue. + +Importance: + Should only be touched if it is a Blocker/Gating issue. If it is, please set to High, and only use Critical if you have found a bug that can take down whole infrastructures. + + +Style guide +----------- + +When creating tasks and other roles for use in Ansible please create then using the YAML dictionary format. + +Example YAML dictionary format: + .. code-block:: yaml + + - name: The name of the tasks + module_name: + thing1: "some-stuff" + thing2: "some-other-stuff" + tags: + - some-tag + - some-other-tag + + +Example **NOT** in YAML dictionary format: + .. code-block:: yaml + + - name: The name of the tasks + module_name: thing1="some-stuff" thing2="some-other-stuff" + tags: + - some-tag + - some-other-tag + + +Usage of the ">" and "|" operators should be limited to Ansible conditionals and command modules such as the ansible ``shell`` module. + + +Issues +------ + +When submitting an issue, or working on an issue please ensure the following criteria are met: + * The description clearly states or describes the original problem or root cause of the problem. + * Include historical information on how the problem was identified. + * Any relevant logs are included. + * If the issue is a bug that needs fixing in a branch other than Master, add the ‘backport potential’ tag TO THE ISSUE (not the PR). + * The provided information should be totally self-contained. External access to web services/sites should not be needed. + * If the issue is needed for a hotfix release, add the 'expedite' label. + * Steps to reproduce the problem if possible. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..8f71f43 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..dfe3553 --- /dev/null +++ b/README.rst @@ -0,0 +1,6 @@ +plugins collection +################## +:tags: openstack, cloud, ansible, plugins +:category: \*nix + +Plugins used to power OpenStack-Ansible and our various roles. diff --git a/actions/config_template.py b/actions/config_template.py new file mode 100644 index 0000000..fdb8d5d --- /dev/null +++ b/actions/config_template.py @@ -0,0 +1,240 @@ +# Copyright 2015, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import ConfigParser +import io +import json +import os +import yaml + +from ansible import errors +from ansible.runner.return_data import ReturnData +from ansible import utils +from ansible.utils import template + + +CONFIG_TYPES = { + 'ini': 'return_config_overrides_ini', + 'json': 'return_config_overrides_json', + 'yaml': 'return_config_overrides_yaml' +} + + +class ActionModule(object): + TRANSFERS_FILES = True + + def __init__(self, runner): + self.runner = runner + + def grab_options(self, complex_args, module_args): + """Grab passed options from Ansible complex and module args. + + :param complex_args: ``dict`` + :param module_args: ``dict`` + :returns: ``dict`` + """ + options = dict() + if complex_args: + options.update(complex_args) + + options.update(utils.parse_kv(module_args)) + return options + + @staticmethod + def return_config_overrides_ini(config_overrides, resultant): + """Returns string value from a modified config file. + + :param config_overrides: ``dict`` + :param resultant: ``str`` || ``unicode`` + :returns: ``str`` + """ + config = ConfigParser.RawConfigParser(allow_no_value=True) + config_object = io.BytesIO(resultant.encode('utf-8')) + config.readfp(config_object) + for section, items in config_overrides.items(): + # If the items value is not a dictionary it is assumed that the + # value is a default item for this config type. + if not isinstance(items, dict): + config.set('DEFAULT', str(section), str(items)) + else: + # Attempt to add a section to the config file passing if + # an error is raised that is related to the section + # already existing. + try: + config.add_section(str(section)) + except (ConfigParser.DuplicateSectionError, ValueError): + pass + for key, value in items.items(): + config.set(str(section), str(key), str(value)) + else: + config_object.close() + + resultant_bytesio = io.BytesIO() + try: + config.write(resultant_bytesio) + return resultant_bytesio.getvalue() + finally: + resultant_bytesio.close() + + def return_config_overrides_json(self, config_overrides, resultant): + """Returns config json + + Its important to note that file ordering will not be preserved as the + information within the json file will be sorted by keys. + + :param config_overrides: ``dict`` + :param resultant: ``str`` || ``unicode`` + :returns: ``str`` + """ + original_resultant = json.loads(resultant) + merged_resultant = self._merge_dict( + base_items=original_resultant, + new_items=config_overrides + ) + return json.dumps( + merged_resultant, + indent=4, + sort_keys=True + ) + + def return_config_overrides_yaml(self, config_overrides, resultant): + """Return config yaml. + + :param config_overrides: ``dict`` + :param resultant: ``str`` || ``unicode`` + :returns: ``str`` + """ + original_resultant = yaml.safe_load(resultant) + merged_resultant = self._merge_dict( + base_items=original_resultant, + new_items=config_overrides + ) + return yaml.safe_dump( + merged_resultant, + default_flow_style=False, + width=1000, + ) + + def _merge_dict(self, base_items, new_items): + """Recursively merge new_items into base_items. + + :param base_items: ``dict`` + :param new_items: ``dict`` + :returns: ``dict`` + """ + for key, value in new_items.iteritems(): + if isinstance(value, dict): + base_items[key] = self._merge_dict( + base_items.get(key, {}), + value + ) + elif isinstance(value, list): + if key in base_items and isinstance(base_items[key], list): + base_items[key].extend(value) + else: + base_items[key] = value + else: + base_items[key] = new_items[key] + return base_items + + def run(self, conn, tmp, module_name, module_args, inject, + complex_args=None, **kwargs): + """Run the method""" + if not self.runner.is_playbook: + raise errors.AnsibleError( + 'FAILED: `config_templates` are only available in playbooks' + ) + + options = self.grab_options(complex_args, module_args) + try: + source = options['src'] + dest = options['dest'] + + config_overrides = options.get('config_overrides', dict()) + config_type = options['config_type'] + assert config_type.lower() in ['ini', 'json', 'yaml'] + except KeyError as exp: + result = dict(failed=True, msg=exp) + return ReturnData(conn=conn, comm_ok=False, result=result) + + source_template = template.template( + self.runner.basedir, + source, + inject + ) + + if '_original_file' in inject: + source_file = utils.path_dwim_relative( + inject['_original_file'], + 'templates', + source_template, + self.runner.basedir + ) + else: + source_file = utils.path_dwim(self.runner.basedir, source_template) + + # Open the template file and return the data as a string. This is + # being done here so that the file can be a vault encrypted file. + resultant = template.template_from_file( + self.runner.basedir, + source_file, + inject, + vault_password=self.runner.vault_pass + ) + + if config_overrides: + type_merger = getattr(self, CONFIG_TYPES.get(config_type)) + resultant = type_merger( + config_overrides=config_overrides, + resultant=resultant + ) + + # Retemplate the resultant object as it may have new data within it + # as provided by an override variable. + template.template_from_string( + basedir=self.runner.basedir, + data=resultant, + vars=inject, + fail_on_undefined=True + ) + + # Access to protected method is unavoidable in Ansible 1.x. + new_module_args = dict( + src=self.runner._transfer_str(conn, tmp, 'source', resultant), + dest=dest, + original_basename=os.path.basename(source), + follow=True, + ) + + module_args_tmp = utils.merge_module_args( + module_args, + new_module_args + ) + + # Remove data types that are not available to the copy module + complex_args.pop('config_overrides') + complex_args.pop('config_type') + + # Return the copy module status. Access to protected method is + # unavoidable in Ansible 1.x. + return self.runner._execute_module( + conn, + tmp, + 'copy', + module_args_tmp, + inject=inject, + complex_args=complex_args + ) + diff --git a/callbacks/profile_tasks.py b/callbacks/profile_tasks.py new file mode 100644 index 0000000..2a4fa39 --- /dev/null +++ b/callbacks/profile_tasks.py @@ -0,0 +1,77 @@ +# The MIT License (MIT) +# +# Copyright (c) 2015, Red Hat, Inc. and others +# Copyright (c) 2015, Rackspace US, Inc. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +# THE SOFTWARE. +# +# ---------------------------------------------------------------------------- +# +# Note that this callback plugin isn't enabled by default. If you'd like to +# enable it, add the following line to ansible.cfg in the 'playbooks' +# directory in this repository: +# +# callback_plugins = plugins/callbacks +# +# Add that line prior to running the playbooks and you will have detailed +# timing information for Ansible tasks right after each playbook finishes +# running. +# +import time + + +class CallbackModule(object): + """ + A plugin for timing tasks + """ + def __init__(self): + self.stats = {} + self.current = None + + def playbook_on_task_start(self, name, is_conditional): + """ + Logs the start of each task + """ + if self.current is not None: + # Record the running time of the last executed task + self.stats[self.current] = time.time() - self.stats[self.current] + + # Record the start time of the current task + self.current = name + self.stats[self.current] = time.time() + + def playbook_on_stats(self, stats): + """ + Prints the timings + """ + # Record the timing of the very last task + if self.current is not None: + self.stats[self.current] = time.time() - self.stats[self.current] + + # Sort the tasks by their running time + results = sorted(self.stats.items(), key=lambda value: value[1], + reverse=True) + + # Just keep the top 10 + results = results[:10] + + # Print the timings + for name, elapsed in results: + print "{0:-<70}{1:->9}".format('{0} '.format(name), + ' {0:.02f}s'.format(elapsed)) diff --git a/dev-requirements.txt b/dev-requirements.txt new file mode 100644 index 0000000..f9f762e --- /dev/null +++ b/dev-requirements.txt @@ -0,0 +1,6 @@ +ansible-lint +ansible>=1.9.1,<2.0.0 + +# this is required for the docs build jobs +sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2 +oslosphinx>=2.5.0 # Apache-2.0 diff --git a/doc/Makefile b/doc/Makefile new file mode 100644 index 0000000..c2d57c8 --- /dev/null +++ b/doc/Makefile @@ -0,0 +1,195 @@ +# Makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +PAPER = +BUILDDIR = build + +# User-friendly check for sphinx-build +ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) +$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) +endif + +# Internal variables. +PAPEROPT_a4 = -D latex_paper_size=a4 +PAPEROPT_letter = -D latex_paper_size=letter +ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source +# the i18n builder cannot share the environment and doctrees with the others +I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source + +.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext + +help: + @echo "Please use \`make ' where is one of" + @echo " html to make standalone HTML files" + @echo " dirhtml to make HTML files named index.html in directories" + @echo " singlehtml to make a single large HTML file" + @echo " pickle to make pickle files" + @echo " json to make JSON files" + @echo " htmlhelp to make HTML files and a HTML help project" + @echo " qthelp to make HTML files and a qthelp project" + @echo " applehelp to make an Apple Help Book" + @echo " devhelp to make HTML files and a Devhelp project" + @echo " epub to make an epub" + @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" + @echo " latexpdf to make LaTeX files and run them through pdflatex" + @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" + @echo " text to make text files" + @echo " man to make manual pages" + @echo " texinfo to make Texinfo files" + @echo " info to make Texinfo files and run them through makeinfo" + @echo " gettext to make PO message catalogs" + @echo " changes to make an overview of all changed/added/deprecated items" + @echo " xml to make Docutils-native XML files" + @echo " pseudoxml to make pseudoxml-XML files for display purposes" + @echo " linkcheck to check all external links for integrity" + @echo " doctest to run all doctests embedded in the documentation (if enabled)" + @echo " coverage to run coverage check of the documentation (if enabled)" + +clean: + rm -rf $(BUILDDIR)/* + +html: + $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." + +dirhtml: + $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." + +singlehtml: + $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml + @echo + @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." + +pickle: + $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle + @echo + @echo "Build finished; now you can process the pickle files." + +json: + $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json + @echo + @echo "Build finished; now you can process the JSON files." + +htmlhelp: + $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp + @echo + @echo "Build finished; now you can run HTML Help Workshop with the" \ + ".hhp project file in $(BUILDDIR)/htmlhelp." + +qthelp: + $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp + @echo + @echo "Build finished; now you can run "qcollectiongenerator" with the" \ + ".qhcp project file in $(BUILDDIR)/qthelp, like this:" + @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/openstack-ansible-plugins.qhcp" + @echo "To view the help file:" + @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/openstack-ansible-plugins.qhc" + +applehelp: + $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp + @echo + @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." + @echo "N.B. You won't be able to view it unless you put it in" \ + "~/Library/Documentation/Help or install it in your application" \ + "bundle." + +devhelp: + $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp + @echo + @echo "Build finished." + @echo "To view the help file:" + @echo "# mkdir -p $$HOME/.local/share/devhelp/openstack-ansible-plugins" + @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/openstack-ansible-plugins" + @echo "# devhelp" + +epub: + $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub + @echo + @echo "Build finished. The epub file is in $(BUILDDIR)/epub." + +latex: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo + @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." + @echo "Run \`make' in that directory to run these through (pdf)latex" \ + "(use \`make latexpdf' here to do that automatically)." + +latexpdf: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through pdflatex..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +latexpdfja: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through platex and dvipdfmx..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +text: + $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text + @echo + @echo "Build finished. The text files are in $(BUILDDIR)/text." + +man: + $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man + @echo + @echo "Build finished. The manual pages are in $(BUILDDIR)/man." + +texinfo: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo + @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." + @echo "Run \`make' in that directory to run these through makeinfo" \ + "(use \`make info' here to do that automatically)." + +info: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo "Running Texinfo files through makeinfo..." + make -C $(BUILDDIR)/texinfo info + @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." + +gettext: + $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale + @echo + @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." + +changes: + $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes + @echo + @echo "The overview file is in $(BUILDDIR)/changes." + +linkcheck: + $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck + @echo + @echo "Link check complete; look for any errors in the above output " \ + "or in $(BUILDDIR)/linkcheck/output.txt." + +doctest: + $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest + @echo "Testing of doctests in the sources finished, look at the " \ + "results in $(BUILDDIR)/doctest/output.txt." + +coverage: + $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage + @echo "Testing of coverage in the sources finished, look at the " \ + "results in $(BUILDDIR)/coverage/python.txt." + +xml: + $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml + @echo + @echo "Build finished. The XML files are in $(BUILDDIR)/xml." + +pseudoxml: + $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml + @echo + @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." + +livehtml: html + sphinx-autobuild -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html diff --git a/doc/source/conf.py b/doc/source/conf.py new file mode 100644 index 0000000..e2c0ed1 --- /dev/null +++ b/doc/source/conf.py @@ -0,0 +1,290 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# +# openstack-ansible-plugins documentation build configuration file, created by +# sphinx-quickstart on Mon Apr 13 20:42:26 2015. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# sys.path.insert(0, os.path.abspath('.')) + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.autodoc', + 'oslosphinx' +] + +# The link to the browsable source code (for the left hand menu) +oslosphinx_cgit_link = 'http://git.openstack.org/cgit/openstack/openstack-ansible-plugins' + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The encoding of source files. +# source_encoding = 'utf-8-sig' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = 'openstack-ansible-plugins' +copyright = '2015, openstack-ansible-plugins contributors' +author = 'openstack-ansible-plugins contributors' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = 'master' +# The full version, including alpha/beta/rc tags. +release = 'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +# today = '' +# Else, today_fmt is used as the format for a strftime call. +# today_fmt = '%B %d, %Y' + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +exclude_patterns = [] + +# The reST default role (used for this markup: `text`) to use for all +# documents. +# default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +# add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +# add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +# show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +# modindex_common_prefix = [] + +# If true, keep warnings as "system message" paragraphs in the built documents. +# keep_warnings = False + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# html_theme = 'alabaster' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# html_theme_options = {} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# " v documentation". +# html_title = None + +# A shorter title for the navigation bar. Default is the same as html_title. +# html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +# html_logo = None + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +# html_favicon = None + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +# Add any extra paths that contain custom files (such as robots.txt or +# .htaccess) here, relative to this directory. These files are copied +# directly to the root of the documentation. +# html_extra_path = [] + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +# html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +# html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +# html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +# html_additional_pages = {} + +# If false, no module index is generated. +# html_domain_indices = True + +# If false, no index is generated. +# html_use_index = True + +# If true, the index is split into individual pages for each letter. +# html_split_index = False + +# If true, links to the reST sources are added to the pages. +# html_show_sourcelink = True + +# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. +# html_show_sphinx = True + +# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. +# html_show_copyright = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +# html_use_opensearch = '' + +# This is the file name suffix for HTML files (e.g. ".xhtml"). +# html_file_suffix = None + +# Language to be used for generating the HTML full-text search index. +# Sphinx supports the following languages: +# 'da', 'de', 'en', 'es', 'fi', 'fr', 'h', 'it', 'ja' +# 'nl', 'no', 'pt', 'ro', 'r', 'sv', 'tr' +# html_search_language = 'en' + +# A dictionary with options for the search language support, empty by default. +# Now only 'ja' uses this config value +# html_search_options = {'type': 'default'} + +# The name of a javascript file (relative to the configuration directory) that +# implements a search results scorer. If empty, the default will be used. +# html_search_scorer = 'scorer.js' + +# Output file base name for HTML help builder. +htmlhelp_basename = 'openstack-ansible-pluginsdoc' + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # 'preamble': '', + + # Latex figure (float) alignment + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, 'openstack-ansible-plugins.tex', + 'openstack-ansible-plugins Documentation', + 'openstack-ansible-plugins contributors', 'manual'), +] + +# The name of an image file (relative to this directory) to place at the top of +# the title page. +# latex_logo = None + +# For "manual" documents, if this is true, then toplevel headings are parts, +# not chapters. +# latex_use_parts = False + +# If true, show page references after internal links. +# latex_show_pagerefs = False + +# If true, show URL addresses after external links. +# latex_show_urls = False + +# Documents to append as an appendix to all manuals. +# latex_appendices = [] + +# If false, no module index is generated. +# latex_domain_indices = True + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, 'openstack-ansible-plugins', + 'openstack-ansible-plugins Documentation', + [author], 1) +] + +# If true, show URL addresses after external links. +# man_show_urls = False + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, 'openstack-ansible-plugins', + 'openstack-ansible-plugins Documentation', + author, 'openstack-ansible-plugins', 'One line description of project.', + 'Miscellaneous'), +] + +# Documents to append as an appendix to all manuals. +# texinfo_appendices = [] + +# If false, no module index is generated. +# texinfo_domain_indices = True + +# How to display URL addresses: 'footnote', 'no', or 'inline'. +# texinfo_show_urls = 'footnote' + +# If true, do not generate a @detailmenu in the "Top" node's menu. +# texinfo_no_detailmenu = False diff --git a/doc/source/index.rst b/doc/source/index.rst new file mode 100644 index 0000000..c82de2e --- /dev/null +++ b/doc/source/index.rst @@ -0,0 +1,36 @@ +plugins Docs +============ + +These are the plugins the OpenStack-Ansible deployment project relies on. +The plugins can be added to any openstack deployment by quite simply cloning +this repository into your plugin and library source and setting up the +``ansible.cfg`` file to point at them as additional plugins for your project. + + +Example ansible.cfg file +------------------------ + +.. code-block:: ini + + [defaults] + lookup_plugins = /etc/ansible/plugins/lookups + filter_plugins = /etc/ansible/plugins/filters + action_plugins = /etc/ansible/plugins/actions + library = /etc/ansible/plugins/library + + +Example role requirement overload for automatic plugin download +--------------------------------------------------------------- + +The Ansible role requirement file can be used to overload the ``ansible-galaxy`` +command to automatically fetch the plugins for you in a given project. To do this +add the following lines to your ``ansible-role-requirements.yml`` file. + +.. code-block:: yaml + + - name: plugins + src: https://github.com/openstack/openstack-ansible-plugins + path: /etc/ansible + scm: git + version: master + diff --git a/filters/osa-filters.py b/filters/osa-filters.py new file mode 100644 index 0000000..76ce8a4 --- /dev/null +++ b/filters/osa-filters.py @@ -0,0 +1,244 @@ +# Copyright 2015, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# (c) 2015, Kevin Carter + +import os +import re +import urlparse +import hashlib + +from ansible import errors + +"""Filter usage: + +Simple filters that may be useful from within the stack +""" + + +def _pip_requirement_split(requirement): + version_descriptors = "(>=|<=|>|<|==|~=|!=)" + requirement = requirement.split(';') + requirement_info = re.split(r'%s\s*' % version_descriptors, requirement[0]) + name = requirement_info[0] + marker = None + if len(requirement) > 1: + marker = requirement[1] + versions = None + if len(requirement_info) > 1: + versions = requirement_info[1] + + return name, versions, marker + + +def _lower_set_lists(list_one, list_two): + + _list_one = set([i.lower() for i in list_one]) + _list_two = set([i.lower() for i in list_two]) + return _list_one, _list_two + + +def bit_length_power_of_2(value): + """Return the smallest power of 2 greater than a numeric value. + + :param value: Number to find the smallest power of 2 + :type value: ``int`` + :returns: ``int`` + """ + return 2**(int(value)-1).bit_length() + + +def get_netloc(url): + """Return the netloc from a URL. + + If the input value is not a value URL the method will raise an Ansible + filter exception. + + :param url: the URL to parse + :type url: ``str`` + :returns: ``str`` + """ + try: + netloc = urlparse.urlparse(url).netloc + except Exception as exp: + raise errors.AnsibleFilterError( + 'Failed to return the netloc of: "%s"' % str(exp) + ) + else: + return netloc + + +def get_netloc_no_port(url): + """Return the netloc without a port from a URL. + + If the input value is not a value URL the method will raise an Ansible + filter exception. + + :param url: the URL to parse + :type url: ``str`` + :returns: ``str`` + """ + return get_netloc(url=url).split(':')[0] + + +def get_netorigin(url): + """Return the netloc from a URL. + + If the input value is not a value URL the method will raise an Ansible + filter exception. + + :param url: the URL to parse + :type url: ``str`` + :returns: ``str`` + """ + try: + parsed_url = urlparse.urlparse(url) + netloc = parsed_url.netloc + scheme = parsed_url.scheme + except Exception as exp: + raise errors.AnsibleFilterError( + 'Failed to return the netorigin of: "%s"' % str(exp) + ) + else: + return '%s://%s' % (scheme, netloc) + + +def string_2_int(string): + """Return the an integer from a string. + + The string is hashed, converted to a base36 int, and the modulo of 10240 + is returned. + + :param string: string to retrieve an int from + :type string: ``str`` + :returns: ``int`` + """ + # Try to encode utf-8 else pass + try: + string = string.encode('utf-8') + except AttributeError: + pass + hashed_name = hashlib.sha256(string).hexdigest() + return int(hashed_name, 36) % 10240 + + +def pip_requirement_names(requirements): + """Return a ``str`` of requirement name and list of versions. + :param requirement: Name of a requirement that may have versions within + it. This will use the constant, + VERSION_DESCRIPTORS. + :type requirement: ``str`` + :return: ``str`` + """ + + named_requirements = list() + for requirement in requirements: + name = _pip_requirement_split(requirement)[0] + if name and not name.startswith('#'): + named_requirements.append(name.lower()) + + return sorted(set(named_requirements)) + + +def pip_constraint_update(list_one, list_two): + + _list_one, _list_two = _lower_set_lists(list_one, list_two) + _list_one, _list_two = list(_list_one), list(_list_two) + for item2 in _list_two: + item2_name, item2_versions, _ = _pip_requirement_split(item2) + if item2_versions: + for item1 in _list_one: + if item2_name == _pip_requirement_split(item1)[0]: + item1_index = _list_one.index(item1) + _list_one[item1_index] = item2 + break + else: + _list_one.append(item2) + + return sorted(_list_one) + + +def splitlines(string_with_lines): + """Return a ``list`` from a string with lines.""" + + return string_with_lines.splitlines() + + +def filtered_list(list_one, list_two): + + _list_one, _list_two = _lower_set_lists(list_one, list_two) + return list(_list_one-_list_two) + + +def git_link_parse(repo): + """Return a dict containing the parts of a git repository. + + :param repo: git repo string to parse. + :type repo: ``str`` + :returns: ``dict`` + """ + + if 'git+' in repo: + _git_url = repo.split('git+', 1)[-1] + else: + _git_url = repo + + if '@' in _git_url: + url, branch = _git_url.split('@', 1) + else: + url = _git_url + branch = 'master' + + name = os.path.basename(url.rstrip('/')) + _branch = branch.split('#') + branch = _branch[0] + + plugin_path = None + # Determine if the package is a plugin type + if len(_branch) > 1 and 'subdirectory=' in _branch[-1]: + plugin_path = _branch[-1].split('subdirectory=')[-1].split('&')[0] + + return { + 'name': name.split('.git')[0].lower(), + 'version': branch, + 'plugin_path': plugin_path, + 'url': url, + 'original': repo + } + + +def git_link_parse_name(repo): + """Return the name of a git repo.""" + + return git_link_parse(repo)['name'] + + +class FilterModule(object): + """Ansible jinja2 filters.""" + + @staticmethod + def filters(): + return { + 'bit_length_power_of_2': bit_length_power_of_2, + 'netloc': get_netloc, + 'netloc_no_port': get_netloc_no_port, + 'netorigin': get_netorigin, + 'string_2_int': string_2_int, + 'pip_requirement_names': pip_requirement_names, + 'pip_constraint_update': pip_constraint_update, + 'splitlines': splitlines, + 'filtered_list': filtered_list, + 'git_link_parse': git_link_parse, + 'git_link_parse_name': git_link_parse_name + } diff --git a/library/config_template b/library/config_template new file mode 100644 index 0000000..0c343aa --- /dev/null +++ b/library/config_template @@ -0,0 +1,66 @@ +# this is a virtual module that is entirely implemented server side + +DOCUMENTATION = """ +--- +module: config_template +version_added: 1.9.2 +short_description: Renders template files providing a create/update override interface +description: + - The module contains the template functionality with the ability to override items + in config, in transit, through the use of a simple dictionary without having to + write out various temp files on target machines. The module renders all of the + potential jinja a user could provide in both the template file and in the override + dictionary which is ideal for deployers who may have lots of different configs + using a similar code base. + - The module is an extension of the **copy** module and all of attributes that can be + set there are available to be set here. +options: + src: + description: + - Path of a Jinja2 formatted template on the local server. This can be a relative + or absolute path. + required: true + default: null + dest: + description: + - Location to render the template to on the remote machine. + required: true + default: null + config_overrides: + description: + - A dictionary used to update or override items within a configuration template. + The dictionary data structure may be nested. If the target config file is an ini + file the nested keys in the ``config_overrides`` will be used as section + headers. + config_type: + description: + - A string value describing the target config type. + choices: + - ini + - json + - yaml +author: Kevin Carter +""" + +EXAMPLES = """ + - name: run config template ini + config_template: + src: templates/test.ini.j2 + dest: /tmp/test.ini + config_overrides: {} + config_type: ini + + - name: run config template json + config_template: + src: templates/test.json.j2 + dest: /tmp/test.json + config_overrides: {} + config_type: json + + - name: run config template yaml + config_template: + src: templates/test.yaml.j2 + dest: /tmp/test.yaml + config_overrides: {} + config_type: yaml +""" diff --git a/library/dist_sort b/library/dist_sort new file mode 100644 index 0000000..d4d1039 --- /dev/null +++ b/library/dist_sort @@ -0,0 +1,168 @@ +#!/usr/bin/env python +# (c) 2014, Kevin Carter +# +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# import module snippets +from ansible.module_utils.basic import * + +DOCUMENTATION = """ +--- +module: dist_sort +version_added: "1.6.6" +short_description: + - Deterministically sort a list to distribute the elements in the list + evenly. Based on external values such as host or static modifier. Returns + a string as named key ``sorted_list``. +description: + - This module returns a list of servers uniquely sorted based on a index + from a look up value location within a group. The group should be an + existing ansible inventory group. This will module returns the sorted + list as a delimited string. +options: + src_list: + description: + - list in the form of a string separated by a delimiter. + required: True + ref_list: + description: + - list to lookup value_to_lookup against to return index number + This should be a pre-determined ansible group containing the + ``value_to_lookup``. + required: False + value_to_lookup: + description: + - value is looked up against ref_list to get index number. + required: False + sort_modifier: + description: + - add a static int into the sort equation to weight the output. + type: int + default: 0 + delimiter: + description: + - delimiter used to parse ``src_list`` with. + default: ',' +author: + - Kevin Carter + - Sam Yaple +""" + +EXAMPLES = """ +- dist_sort: + value_to_lookup: "Hostname-in-ansible-group_name" + ref_list: "{{ groups['group_name'] }}" + src_list: "Server1,Server2,Server3" + register: test_var + +# With a pre-set delimiter +- dist_sort: + value_to_lookup: "Hostname-in-ansible-group_name" + ref_list: "{{ groups['group_name'] }}" + src_list: "Server1|Server2|Server3" + delimiter: '|' + register: test_var + +# With a set modifier +- dist_sort: + value_to_lookup: "Hostname-in-ansible-group_name" + ref_list: "{{ groups['group_name'] }}" + src_list: "Server1#Server2#Server3" + delimiter: '#' + sort_modifier: 5 + register: test_var +""" + + +class DistSort(object): + def __init__(self, module): + """Deterministically sort a list of servers. + + :param module: The active ansible module. + :type module: ``class`` + """ + self.module = module + self.params = self.module.params + self.return_data = self._runner() + + def _runner(self): + """Return the sorted list of servers. + + Based on the modulo of index of a *value_to_lookup* from an ansible + group this function will return a comma "delimiter" separated list of + items. + + :returns: ``str`` + """ + index = self.params['ref_list'].index(self.params['value_to_lookup']) + index += self.params['sort_modifier'] + src_list = self.params['src_list'].split( + self.params['delimiter'] + ) + + for _ in range(index % len(src_list)): + src_list.append(src_list.pop(0)) + else: + return self.params['delimiter'].join(src_list) + + +def main(): + """Run the main app.""" + module = AnsibleModule( + argument_spec=dict( + value_to_lookup=dict( + required=True, + type='str' + ), + ref_list=dict( + required=True, + type='list' + ), + src_list=dict( + required=True, + type='str' + ), + delimiter=dict( + required=False, + type='str', + default=',' + ), + sort_modifier=dict( + required=False, + type='str', + default='0' + ) + ), + supports_check_mode=False + ) + try: + # This is done so that the failure can be parsed and does not cause + # ansible to fail if a non-int is passed. + module.params['sort_modifier'] = int(module.params['sort_modifier']) + + _ds = DistSort(module=module) + if _ds.return_data == module.params['src_list']: + _changed = False + else: + _changed = True + + module.exit_json(changed=_changed, **{'sorted_list': _ds.return_data}) + except Exception as exp: + resp = {'stderr': str(exp)} + resp.update(module.params) + module.fail_json(msg='Failed Process', **resp) + +if __name__ == '__main__': + main() diff --git a/library/glance b/library/glance new file mode 100644 index 0000000..73e5997 --- /dev/null +++ b/library/glance @@ -0,0 +1,236 @@ +#!/usr/bin/env python +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import glanceclient.client as glclient +import keystoneclient.v3.client as ksclient + +# import module snippets +from ansible.module_utils.basic import * + + +DOCUMENTATION = """ +--- +module: glance +short_description: + - Basic module for interacting with openstack glance +description: + - Basic module for interacting with openstack glance +options: + command: + description: + - Operation for the module to perform. Currently available + choices: + - image-list + - image-create + openrc_path: + decription: + - Path to openrc file from which credentials and keystoneclient + - endpoint will be extracted + image_name: + description: + - Name of the image to create + image_url: + description: + - URL from which to download the image data + image_container_format: + description: + - container format that the image uses (bare) + image_disk_format: + description: + - disk format that the image uses + image_is_public: + description: + - Should the image be visible to all tenants? + choices: + - true (public) + - false (private) + api_version: + description: + - which version of the glance api to use + choices: + - 1 + - 2 + default: 1 + insecure: + description: + - Explicitly allow client to perform "insecure" TLS + choices: + - false + - true + default: false +author: Hugh Saunders +""" + +EXAMPLES = """ +# Create an image +- name: Ensure cirros image + glance: + command: 'image-create' + openrc_path: /root/openrc + image_name: cirros + image_url: 'https://example-domain.com/cirros-0.3.2-source.tar.gz' + image_container_format: bare + image_disk_format: qcow2 + image_is_public: True + +# Get facts about existing images +- name: Get image facts + glance: + command: 'image-list' + openrc_path: /root/openrc +""" + + +COMMAND_MAP = {'image-list': 'list_images', + 'image-create': 'create_image'} + + +class ManageGlance(object): + def __init__(self, module): + self.state_change = False + self.glance = None + self.keystone = None + self.module = module + try: + self._keystone_authenticate() + self._init_glance() + except Exception as e: + self.module.fail_json( + err="Initialisation Error: %s" % e, + rc=2, msg=str(e)) + + def _parse_openrc(self): + """Get credentials from an openrc file.""" + openrc_path = self.module.params['openrc_path'] + line_re = re.compile('^export (?POS_\w*)=(?P[^\n]*)') + with open(openrc_path) as openrc: + matches = [line_re.match(l) for l in openrc] + return dict( + (g.groupdict()['key'], g.groupdict()['value']) + for g in matches if g + ) + + def _keystone_authenticate(self): + """Authenticate with Keystone.""" + openrc = self._parse_openrc() + insecure = self.module.params['insecure'] + self.keystone = ksclient.Client(insecure=insecure, + username=openrc['OS_USERNAME'], + password=openrc['OS_PASSWORD'], + project_name=openrc['OS_PROJECT_NAME'], + auth_url=openrc['OS_AUTH_URL']) + + def _init_glance(self): + """Create glance client object using token and url from keystone.""" + openrc = self._parse_openrc() + p = self.module.params + v = p['api_version'] + ep = self.keystone.service_catalog.url_for( + service_type='image', + endpoint_type=openrc['OS_ENDPOINT_TYPE'] + ) + + self.glance = glclient.Client( + endpoint='%s/v%s' % (ep, v), + token=self.keystone.get_token(self.keystone.session) + ) + + def route(self): + """Run the command specified by the command parameter.""" + getattr(self, COMMAND_MAP[self.module.params['command']])() + + def _get_image_facts(self): + """Helper function to format image list as a dictionary.""" + p = self.module.params + v = p['api_version'] + if v == '1': + return dict( + (i.name, i.to_dict()) for i in self.glance.images.list() + ) + elif v == '2': + return dict( + (i.name, i) for i in self.glance.images.list() + ) + + def list_images(self): + """Get information about available glance images. + + Returns as a fact dictionary glance_images + """ + self.module.exit_json( + changed=self.state_change, + ansible_facts=dict(glance_images=self._get_image_facts())) + + def create_image(self): + """Create a glance image that references a remote url.""" + p = self.module.params + v = p['api_version'] + image_name = p['image_name'] + image_opts = dict( + name=image_name, + disk_format=p['image_disk_format'], + container_format=p['image_container_format'], + copy_from=p['image_url'] + ) + if v == '1': + image_opts['is_public'] = p['image_is_public'] + elif v == '2': + if p['image_is_public']: + vis = 'public' + else: + vis = 'private' + image_opts['visibility'] = vis + + images = {i.name for i in self.glance.images.list()} + if image_name in images: + self.module.exit_json( + changed=self.state_change, + ansible_facts=dict( + glance_images=self._get_image_facts() + ) + ) + else: + self.glance.images.create(**image_opts) + self.state_change = True + self.module.exit_json( + changed=self.state_change, + ansible_facts=dict( + glance_images=self._get_image_facts() + ) + ) + + +def main(): + module = AnsibleModule( + argument_spec=dict( + command=dict(required=True, choices=COMMAND_MAP.keys()), + openrc_path=dict(required=True), + image_name=dict(required=False), + image_url=dict(required=False), + image_container_format=dict(required=False), + image_disk_format=dict(required=False), + image_is_public=dict(required=False, choices=BOOLEANS), + api_version=dict(default='1', required=False, choices=['1', '2']), + insecure=dict(default=False, required=False, + choices=BOOLEANS + ['True', 'False']) + ), + supports_check_mode=False + ) + mg = ManageGlance(module) + mg.route() + +if __name__ == '__main__': + main() diff --git a/library/keystone b/library/keystone new file mode 100644 index 0000000..e9d04be --- /dev/null +++ b/library/keystone @@ -0,0 +1,1309 @@ +#!/usr/bin/python +# (c) 2014, Kevin Carter +# +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Based on Jimmy Tang's implementation + +DOCUMENTATION = """ +--- +module: keystone +version_added: "1.6.2" +short_description: + - Manage OpenStack Identity (keystone) users, projects, roles, and + endpoints. +description: + - Manage OpenStack Identity (keystone) users, projects, roles, and + endpoints. +options: + return_code: + description: + - Allow for return Codes other than 0 when executing commands. + - This is a comma separated list of acceptable return codes. + default: 0 + login_user: + description: + - login username to authenticate to keystone + required: false + default: admin + login_password: + description: + - Password of login user + required: false + default: 'yes' + login_project_name: + description: + - The project login_user belongs to + required: false + default: None + login_tenant_name: + description: + - The tenant login_user belongs to + required: false + default: None + token: + description: + - The token to be uses in case the password is not specified + required: false + default: None + endpoint: + description: + - The keystone url for authentication + required: false + password: + description: + - The password to be assigned to the user + required: false + default: None + user_name: + description: + - The name of the user that has to added/removed from OpenStack + required: false + default: None + project_name: + description: + - The project name that has be added/removed + required: false + default: None + tenant_name: + description: + - The tenant name that has be added/removed + required: false + default: None + role_name: + description: + - The name of the role to be assigned or created + required: false + service_name: + description: + - Name of the service. + required: false + default: None + region_name: + description: + - Name of the region. + required: false + default: None + domain_name: + description: + - Name of the domain to add a project to. + required: false + default: 'Default' + description: + description: + - A description for the project + required: false + default: None + email: + description: + - Email address for the user, this is only used in "ensure_user" + required: false + default: None + service_type: + description: + - Type of service. + required: false + default: None + endpoint_list: + description: + - List of endpoints to add to keystone for a service + required: false + default: None + type: list + group_name: + description: + - A name for the group + required: False + default: None + idp_name: + description: + - A name for the identity provider + required: False + default: None + idp_remote_ids: + description: + - A URL that identifies the remote identity provider + required: False + default: None + idp_enabled: + description: + - Set whether a remote identity provider is enabled + required: False + default: True + sp_name: + description: + - A name for the service provider + required: False + default: None + sp_enabled: + description: + - Set whether a service provider is enabled + required: False + default: True + sp_url: + description: + - URL where the service provider expects to receive SAML assertions + - eg: http(s)://${SP_HOST}:5000/Shibboleth.sso/SAML2/ECP + required: False + default: None + sp_auth_url: + description: + - URL for federated users to request tokens from + - eg: http(s)://${SP_HOST}:5000/v3/OS-FEDERATION + /identity_providers/${IDP_ID}/saml2/auth + required: False + default: None + protocol_name: + description: + - A name for the protocol + required: False + default: None + mapping_name: + description: + - A name for the mapping + required: False + default: None + mapping_rules: + description: + - A dictionary mapping federated users to local groups. + - see: http://specs.openstack.org/openstack/keystone-specs + /api/v3/identity-api-v3-os-federation-ext.html#mappings + required: False + default: None + domain_enabled: + description: + - Name for a domain + required: False + default: True + command: + description: + - Indicate desired state of the resource + choices: ['get_tenant', 'get_project', 'get_user', 'get_role', + 'ensure_service', 'ensure_endpoint', 'ensure_role', + 'ensure_user', 'ensure_user_role', 'ensure_tenant', + 'ensure_project', 'ensure_service_provider', + 'ensure_group', 'ensure_identity_provider', + 'ensure_protocol', ensure_mapping', + 'ensure_group_role'] + required: true + insecure: + description: + - Explicitly allow client to perform "insecure" TLS + choices: + - false + - true + default: false +requirements: [ python-keystoneclient ] +author: Kevin Carter +""" + +EXAMPLES = """ +# Create an admin project +- keystone: + command: "ensure_project" + project_name: "admin" + domain_name: "Default" + description: "Admin project" + +# Create a service project +- keystone: + command: "ensure_project" + project_name: "service" + description: "Service project" + +# Create an admin user +- keystone: + command: "ensure_user" + user_name: "admin" + project_name: "admin" + password: "secrete" + email: "admin@some-domain.com" + +# Create an admin role +- keystone: + command: "ensure_role" + role_name: "admin" + +# Create a user +- keystone: + command: "ensure_user" + user_name: "glance" + project_name: "service" + password: "secrete" + domain_name: "Default" + email: "glance@some-domain.com" + +# Add a role to a user +- keystone: + command: "ensure_user_role" + user_name: "glance" + project_name: "service" + role_name: "admin" + +# Add a project role to a group +- keystone: + command: "ensure_group_role" + group_name: "fedgroup" + project_name: "fedproject" + role_name: "_member_" + +# Create a service +- keystone: + command: "ensure_service" + service_name: "glance" + service_type: "image" + description: "Glance Image Service" + +# Create an endpoint +- keystone: + command: "ensure_endpoint" + region_name: "RegionOne" + service_name: "glance" + service_type: "image" + endpoint_list: + - url: "http://127.0.0.1:9292" + interface: "public" + - url: "http://127.0.0.1:9292" + interface: "admin" + - url: "http://127.0.0.1:9292" + interface: "internal" + +# Get project id +- keystone: + command: "get_project" + project_name: "admin" + +# Get user id +- keystone: + command: "get_user" + user_name: "admin" + +# Get role id +- keystone: + command: "get_role" + user_name: "admin" + +""" + +COMMAND_MAP = { + 'get_tenant': { + 'variables': [ + 'project_name', + 'tenant_name' + ] + }, + 'get_project': { + 'variables': [ + 'project_name', + 'tenant_name' + ] + }, + 'get_user': { + 'variables': [ + 'user_name' + ] + }, + 'get_role': { + 'variables': [ + 'role_name', + 'project_name', + 'tenant_name', + 'user_name' + ] + }, + 'ensure_service': { + 'variables': [ + 'service_name', + 'service_type', + 'description' + ] + }, + 'ensure_endpoint': { + 'variables': [ + 'region_name', + 'service_name', + 'service_type', + 'endpoint_list' + ] + }, + 'ensure_role': { + 'variables': [ + 'role_name' + ] + }, + 'ensure_user': { + 'variables': [ + 'project_name', + 'tenant_name', + 'user_name', + 'password', + 'email', + 'domain_name' + ] + }, + 'ensure_user_role': { + 'variables': [ + 'user_name', + 'project_name', + 'tenant_name', + 'role_name' + ] + }, + 'ensure_group_role': { + 'variables': [ + 'group_name', + 'project_name', + 'role_name' + ] + }, + 'ensure_project': { + 'variables': [ + 'project_name', + 'tenant_name', + 'description', + 'domain_name' + ] + }, + 'ensure_tenant': { + 'variables': [ + 'project_name', + 'tenant_name', + 'description', + 'domain_name' + ] + }, + 'ensure_group': { + 'variables': [ + 'group_name', + 'domain_name' + ] + }, + 'ensure_identity_provider': { + 'variables': [ + 'idp_name', + 'idp_remote_ids', + 'idp_enabled' + ] + }, + 'ensure_service_provider': { + 'variables': [ + 'sp_name', + 'sp_url', + 'sp_auth_url', + 'sp_enabled' + ] + }, + 'ensure_protocol': { + 'variables': [ + 'protocol_name', + 'idp_name', + 'mapping_name' + ] + }, + 'ensure_mapping': { + 'variables': [ + 'mapping_name', + 'mapping_rules', + ] + }, + 'ensure_domain': { + 'variables': [ + 'domain_name', + 'domain_enabled' + ] + } +} + +try: + from keystoneclient import exceptions as kexceptions + from keystoneclient.v3 import client +except ImportError: + keystoneclient_found = False +else: + keystoneclient_found = True + + +class ManageKeystone(object): + def __init__(self, module): + """Manage Keystone via Ansible.""" + self.state_change = False + self.keystone = None + + # Load AnsibleModule + self.module = module + + def command_router(self): + """Run the command as its provided to the module.""" + command_name = self.module.params['command'] + if command_name not in COMMAND_MAP: + self.failure( + error='No Command Found', + rc=2, + msg='Command [ %s ] was not found.' % command_name + ) + + action_command = COMMAND_MAP[command_name] + if hasattr(self, '%s' % command_name): + action = getattr(self, '%s' % command_name) + facts = action(variables=action_command['variables']) + if facts is None: + self.module.exit_json(changed=self.state_change) + else: + self.module.exit_json( + changed=self.state_change, + ansible_facts=facts + ) + else: + self.failure( + error='Command not in ManageKeystone class', + rc=2, + msg='Method [ %s ] was not found.' % command_name + ) + + @staticmethod + def _facts(facts): + """Return a dict for our Ansible facts. + + :param facts: ``dict`` Dict with data to return + """ + return {'keystone_facts': facts} + + def _get_vars(self, variables, required=None): + """Return a dict of all variables as found within the module. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + :param required: ``list`` Name of variables that are required. + """ + return_dict = {} + for variable in variables: + return_dict[variable] = self.module.params.get(variable) + else: + if isinstance(required, list): + for var_name in required: + check = return_dict.get(var_name) + if check is None: + self.failure( + error='Missing [ %s ] from Task or found a None' + ' value' % var_name, + rc=000, + msg='variables %s - available params [ %s ]' + % (variables, self.module.params) + ) + return return_dict + + def failure(self, error, rc, msg): + """Return a Failure when running an Ansible command. + + :param error: ``str`` Error that occurred. + :param rc: ``int`` Return code while executing an Ansible command. + :param msg: ``str`` Message to report. + """ + self.module.fail_json(msg=msg, rc=rc, err=error) + + def _authenticate(self): + """Return a keystone client object.""" + required_vars = ['endpoint'] + variables = [ + 'endpoint', + 'login_user', + 'login_password', + 'login_project_name', + 'login_tenant_name', + 'token', + 'insecure' + ] + variables_dict = self._get_vars(variables, required=required_vars) + + endpoint = variables_dict.pop('endpoint') + login_user = variables_dict.pop('login_user') + login_password = variables_dict.pop('login_password') + login_project_name = (variables_dict.pop('login_project_name', None) or + variables_dict.pop('login_tenant_name')) + token = variables_dict.pop('token') + insecure = variables_dict.pop('insecure') + + if token is None: + if login_project_name is None: + self.failure( + error='Missing Project Name', + rc=2, + msg='If you do not specify a token you must use a project' + ' name for authentication. Try adding' + ' [ login_project_name ] to the task' + ) + if login_password is None: + self.failure( + error='Missing Password', + rc=2, + msg='If you do not specify a token you must use a password' + ' name for authentication. Try adding' + ' [ login_password ] to the task' + ) + + if token: + self.keystone = client.Client( + insecure=insecure, + endpoint=endpoint, + token=token + ) + else: + self.keystone = client.Client( + insecure=insecure, + auth_url=endpoint, + username=login_user, + password=login_password, + project_name=login_project_name + ) + + def _get_domain_from_vars(self, variables): + # NOTE(sigmavirus24): Since we don't require domain, this will be None + # in the dictionary. When we pop it, we can't provide a default + # because 'domain' exists and is None. In order to use a default + # value, we need to use `or 'default'` here to make sure we default to + # the default domain. If we don't do it this way, Keystone throws a + # 401 Unauthorized which is just plain wrong. + domain_name = variables.pop('domain_name', None) or 'Default' + + return self._get_domain(name=domain_name) + + def _get_domain(self, name): + """Return domain information. + + :param str name: Name of the domain. + """ + for entry in self.keystone.domains.list(): + if entry.name == name: + return entry + else: + return None + + def _get_project(self, name): + """Return project information. + + Formerly, _get_tenant + + :param name: ``str`` Name of the project. + """ + for entry in self.keystone.projects.list(): + if entry.name == name: + return entry + else: + return None + + def get_tenant(self, variables): + return self.get_project(variables) + + def get_project(self, variables): + """Return a project id. + + This will return `None` if the ``name`` is not found. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + self._authenticate() + variables_dict = self._get_vars(variables) + project_name = (variables_dict.pop('project_name', None) or + variables_dict.pop('tenant_name')) + project = self._get_project(name=project_name) + if project is None: + self.failure( + error='project [ %s ] was not found.' % project_name, + rc=2, + msg='project was not found, does it exist?' + ) + + return self._facts(facts={'id': project.id}) + + def ensure_tenant(self, variables): + return self.ensure_project(variables) + + def ensure_project(self, variables): + """Create a new project within Keystone if it does not exist. + + Returns the project ID on a successful run. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + self._authenticate() + variables_dict = self._get_vars(variables) + project_name = (variables_dict.pop('project_name', None) or + variables_dict.pop('tenant_name')) + project_description = variables_dict.pop('description') + if project_description is None: + project_description = 'Project %s' % project_name + + domain = self._get_domain_from_vars(variables_dict) + project = self._get_project(name=project_name) + if project is None: + self.state_change = True + project = self.keystone.projects.create( + name=project_name, + description=project_description, + domain=domain, + enabled=True + ) + + return self._facts(facts={'id': project.id}) + + def _get_user(self, name, domain): + """Return a user information. + + This will return `None` if the ``name`` is not found. + + :param name: ``str`` Name of the user. + """ + for entry in self.keystone.users.list(domain=domain): + if getattr(entry, 'name', None) == name: + return entry + else: + return None + + def get_user(self, variables): + """Return a project id. + + This will return `None` if the ``name`` is not found. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + self._authenticate() + variables_dict = self._get_vars(variables, required=['user_name']) + user_name = variables_dict.pop('user_name') + domain = self._get_domain_from_vars(variables_dict) + user = self._get_user(name=user_name, domain=domain) + if user is None: + self.failure( + error='user [ %s ] was not found.' % user_name, + rc=2, + msg='user was not found, does it exist?' + ) + + return self._facts(facts={'id': user.id}) + + def ensure_user(self, variables): + """Create a new user within Keystone if it does not exist. + + Returns the user ID on a successful run. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + self._authenticate() + required_vars = ['user_name', 'password'] + variables_dict = self._get_vars(variables, required=required_vars) + project_name = (variables_dict.pop('project_name', None) or + variables_dict.pop('tenant_name')) + password = variables_dict.pop('password') + user_name = variables_dict.pop('user_name') + email = variables_dict.pop('email') + + domain = self._get_domain_from_vars(variables_dict) + project = self._get_project(name=project_name) + if project is None: + self.failure( + error='project [ %s ] was not found.' % project_name, + rc=2, + msg='project was not found, does it exist?' + ) + + user = self._get_user(name=user_name, domain=domain) + if user is None: + self.state_change = True + user = self.keystone.users.create( + name=user_name, + password=password, + email=email, + domain=domain, + default_project=project + ) + + return self._facts(facts={'id': user.id}) + + def _get_role(self, name, domain): + """Return a role by name. + + This will return `None` if the ``name`` is not found. + + :param name: ``str`` Name of the role. + :param domain: ``str`` ID of the domain + """ + for entry in self.keystone.roles.list(domain=domain): + if entry.name == name: + return entry + else: + return None + + def _get_group(self, name, domain='Default'): + """Return a group by name. + + This will return `None` if the ``name`` is not found. + + :param name: ``str`` Name of the role. + """ + for entry in self.keystone.groups.list(domain=domain): + if domain is None: + if entry.name == name: + return entry + else: + if entry.name == name and entry.domain_id == domain.id: + return entry + else: + return None + + def get_role(self, variables): + """Return a role by name. + + This will return `None` if the ``name`` is not found. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + self._authenticate() + variables_dict = self._get_vars(variables, required=['role_name']) + role_name = variables_dict.pop('role_name') + domain = self._get_domain_from_vars(variables_dict) + role_data = self._get_role(name=role_name, domain=domain) + if role_data is None: + self.failure( + error='role [ %s ] was not found.' % role_name, + rc=2, + msg='role was not found, does it exist?' + ) + + return self._facts(facts={'id': role_data.id}) + + def _get_role_data(self, user_name, project_name, role_name, group_name, + domain): + if user_name is not None: + user = self._get_user(name=user_name, domain=domain) + if user is None: + self.failure( + error='user [ %s ] was not found.' % user_name, + rc=2, + msg='User was not found, does it exist?' + ) + else: + user = None + + project = self._get_project(name=project_name) + if project is None: + self.failure( + error='project [ %s ] was not found.' % project_name, + rc=2, + msg='project was not found, does it exist?' + ) + + role = self._get_role(name=role_name, domain=domain) + if role is None: + self.failure( + error='role [ %s ] was not found.' % role_name, + rc=2, + msg='role was not found, does it exist?' + ) + + if group_name is not None: + group = self._get_group(name=group_name, domain=domain) + if group is None: + self.failure( + error='group [ %s ] was not found.' % group_name, + rc=2, + msg='group was not found, does it exist?' + ) + else: + group = None + + return user, project, role, group + + def ensure_role(self, variables): + """Create a new role within Keystone if it does not exist. + + Returns the user ID on a successful run. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + self._authenticate() + variables_dict = self._get_vars(variables, required=['role_name']) + domain = self._get_domain_from_vars(variables_dict) + role_name = variables_dict.pop('role_name') + + role = self._get_role(name=role_name, domain=domain) + if role is None: + self.state_change = True + role = self.keystone.roles.create(role_name) + + return self._facts(facts={'id': role.id}) + + def _get_user_roles(self, name, user, project): + role_list = self.keystone.roles.list( + user=user, + project=project + ) + for entry in role_list: + if entry.name == name: + return entry + else: + return None + + def _get_group_roles(self, name, group, project, domain): + group_list = self.keystone.roles.list( + group=group, + project=project, + domain=domain + ) + for entry in group_list: + if entry.name == name: + return entry + else: + return None + + def ensure_user_role(self, variables): + self._authenticate() + required_vars = ['user_name', 'role_name'] + variables_dict = self._get_vars(variables, required=required_vars) + domain = self._get_domain_from_vars(variables_dict) + user_name = variables_dict.pop('user_name') + # NOTE(sigmavirus24): Try to get the project_name, but + # don't error out on it. This will change when the playbooks are + # updated to use project_name instead of tenant_name + project_name = (variables_dict.pop('project_name', None) or + variables_dict.pop('tenant_name')) + role_name = variables_dict.pop('role_name') + + user, project, role, group = self._get_role_data( + user_name=user_name, project_name=project_name, + role_name=role_name, group_name=None, domain=domain + ) + + user_role = self._get_user_roles( + name=role_name, user=user, project=project + ) + + if user_role is None: + self.keystone.roles.grant( + user=user, role=role, project=project + ) + user_role = self._get_user_roles( + name=role_name, user=user, project=project + ) + + return self._facts(facts={'id': user_role.id}) + + def ensure_group_role(self, variables): + self._authenticate() + required_vars = ['group_name', 'project_name', 'role_name'] + variables_dict = self._get_vars(variables, required=required_vars) + domain = self._get_domain_from_vars(variables_dict) + group_name = variables_dict.pop('group_name') + project_name = variables_dict.pop('project_name') + role_name = variables_dict.pop('role_name') + + user, project, role, group = self._get_role_data( + group_name=group_name, project_name=project_name, + role_name=role_name, user_name=None, domain=domain + ) + + group_role = self._get_group_roles( + name=role_name, group=group, project=project, domain=domain + ) + + if group_role is None: + self.keystone.roles.grant( + group=group, role=role, project=project + ) + group_role = self._get_group_roles( + name=role_name, + group=group, + project=project, + domain=domain + ) + + return self._facts(facts={'id': group_role.id}) + + def ensure_group(self, variables): + """Create a new group within Keystone if it does not exist. + + Returns the group ID on a successful run. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + + self._authenticate() + required_vars = ['group_name', 'domain_name'] + variables_dict = self._get_vars(variables, required=required_vars) + group_name = variables_dict.pop('group_name') + + domain = self._get_domain_from_vars(variables_dict) + + group = self._get_group( + name=group_name, domain=domain + ) + + if group is None: + self.state_change = True + group = self.keystone.groups.create( + name=group_name, domain=domain + ) + + return self._facts(facts={'id': group.id}) + + def _get_service(self, name, srv_type=None): + for entry in self.keystone.services.list(): + if srv_type is not None: + if entry.type == srv_type and name == entry.name: + return entry + elif entry.name == name: + return entry + else: + return None + + def ensure_service(self, variables): + """Create a new service within Keystone if it does not exist. + + Returns the service ID on a successful run. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + self._authenticate() + required_vars = ['service_name', 'service_type'] + variables_dict = self._get_vars(variables, required=required_vars) + + service_name = variables_dict.pop('service_name') + description = variables_dict.pop('description') + service_type = variables_dict.pop('service_type') + + service = self._get_service(name=service_name, srv_type=service_type) + if service is None or service.type != service_type: + self.state_change = True + service = self.keystone.services.create( + name=service_name, + type=service_type, + description=description + ) + + return self._facts(facts={'id': service.id}) + + def _get_endpoint(self, region, url, interface): + for entry in self.keystone.endpoints.list(): + check = [ + entry.region == region, + entry.url == url, + entry.interface == interface + ] + if all(check): + return entry + else: + return None + + def ensure_endpoint(self, variables): + """Create a new endpoint within Keystone if it does not exist. + + Returns the endpoint ID on a successful run. + + :param variables: ``list`` List of all variables that are available to + use within the Keystone Command. + """ + self._authenticate() + required_vars = [ + 'region_name', + 'service_name', + 'service_type', + 'endpoint_list' + ] + variables_dict = self._get_vars(variables, required=required_vars) + + service_name = variables_dict.pop('service_name') + service_type = variables_dict.pop('service_type') + region = variables_dict.pop('region_name') + endpoint_list = variables_dict.pop('endpoint_list') + + service = self._get_service(name=service_name, srv_type=service_type) + if service is None: + self.failure( + error='service [ %s ] was not found.' % service_name, + rc=2, + msg='Service was not found, does it exist?' + ) + + endpoints = {} + for endpoint_dict in endpoint_list: + url = endpoint_dict.pop('url') + interface = endpoint_dict.pop('interface') + endpoint = self._get_endpoint( + region=region, + url=url, + interface=interface + ) + if endpoint is None: + self.state_change = True + endpoint = self.keystone.endpoints.create( + region=region, + service=service, + url=url, + interface=interface + ) + endpoints[interface] = endpoint + + return self._facts( + facts={'%sid' % interface: endpoint.id + for interface, endpoint in endpoints.items()}) + + def _ensure_generic(self, manager, required_vars, variables): + """Try and create a new 'thing' in keystone. + + Thing type is determined by the manager passed in. + + :param: manager - openstack object manager eg self.keystone.groups + :param: required_vars - dictionary: + ansible module argument name : manager argument name + eg {'group_name': 'name'} + + :returns: Facts dictionary with things = + + + TODO: make this handle updates as well as creates + TODO (maybe, if we decide to use this module long term): + migrate other ensures to use this + """ + + # Get values for variables + variables_dict = self._get_vars(variables, + required=required_vars.keys()) + + # Translate ansible module argument names to manager expected names + args_dict = {required_vars[k]: v for k, v in variables_dict.items()} + + try: + manager.create(**args_dict) + self.state_change = True + except kexceptions.Conflict: + self.state_change = False + + try: + return self._facts(facts={ + manager.collection_key: + [x.to_dict() for x in manager.list()] + }) + except TypeError: + # some managers require arguments to their list functions :/ + # return no facts in this case. + return self._facts(facts={}) + + def ensure_identity_provider(self, variables): + self._authenticate() + return self._ensure_generic( + manager=self.keystone.federation.identity_providers, + required_vars={'idp_name': 'id', + 'idp_remote_ids': 'remote_ids', + 'idp_enabled': 'enabled'}, + variables=variables + ) + + def ensure_service_provider(self, variables): + self._authenticate() + return self._ensure_generic( + manager=self.keystone.federation.service_providers, + required_vars={'sp_name': 'id', + 'sp_auth_url': 'auth_url', + 'sp_url': 'sp_url', + 'sp_enabled': 'enabled'}, + variables=variables + ) + + def ensure_protocol(self, variables): + """Facts not returned + + This is because you can't list protocols without + specifying an identity provider + """ + + self._authenticate() + return self._ensure_generic( + manager=self.keystone.federation.protocols, + required_vars={'protocol_name': 'protocol_id', + 'idp_name': 'identity_provider', + 'mapping_name': 'mapping'}, + variables=variables + ) + + def ensure_mapping(self, variables): + self._authenticate() + return self._ensure_generic( + manager=self.keystone.federation.mappings, + required_vars={'mapping_name': 'mapping_id', + 'mapping_rules': 'rules'}, + variables=variables + ) + + def ensure_domain(self, variables): + self._authenticate() + return self._ensure_generic( + manager=self.keystone.domains, + required_vars={'domain_name': 'name', + 'domain_enabled': 'enabled'}, + variables=variables + ) + + +def main(): + module = AnsibleModule( + argument_spec=dict( + login_user=dict( + required=False + ), + login_password=dict( + required=False + ), + login_tenant_name=dict( + required=False + ), + token=dict( + required=False + ), + password=dict( + required=False + ), + endpoint=dict( + required=True, + ), + user_name=dict( + required=False + ), + tenant_name=dict( + required=False + ), + project_name=dict( + required=False + ), + domain_name=dict( + required=False + ), + role_name=dict( + required=False + ), + service_name=dict( + required=False + ), + region_name=dict( + required=False + ), + description=dict( + required=False + ), + email=dict( + required=False + ), + service_type=dict( + required=False + ), + endpoint_list=dict( + required=False, + type='list' + ), + command=dict( + required=True, + choices=COMMAND_MAP.keys() + ), + insecure=dict( + default=False, + required=False, + choices=BOOLEANS + ['True', 'False'] + ), + return_code=dict( + type='str', + default='0' + ), + group_name=dict( + type='str', + required=False + ), + idp_remote_ids=dict( + type='list', + required=False, + ), + idp_name=dict( + type='str', + required=False, + ), + idp_enabled=dict( + type='bool', + default=True, + required=False, + ), + sp_name=dict( + type='str', + required=False, + ), + sp_auth_url=dict( + type='str', + required=False, + ), + sp_url=dict( + type='str', + required=False, + ), + sp_enabled=dict( + type='bool', + default=True, + required=False, + ), + protocol_name=dict( + type='str', + required=False, + ), + mapping_name=dict( + type='str', + required=False, + ), + mapping_rules=dict( + type='list', + required=False, + ), + domain_enabled=dict( + type='bool', + required=False, + default=True + ) + ), + supports_check_mode=False, + mutually_exclusive=[ + ['token', 'login_user'], + ['token', 'login_password'], + ['token', 'login_tenant_name'] + ] + ) + + km = ManageKeystone(module=module) + if not keystoneclient_found: + km.failure( + error='python-keystoneclient is missing', + rc=2, + msg='keystone client was not importable, is it installed?' + ) + + return_code = module.params.get('return_code', '').split(',') + module.params['return_code'] = return_code + km.command_router() + + +# import module snippets +from ansible.module_utils.basic import * # NOQA +if __name__ == '__main__': + main() diff --git a/library/memcached b/library/memcached new file mode 100644 index 0000000..5208fdd --- /dev/null +++ b/library/memcached @@ -0,0 +1,598 @@ +#!/usr/bin/python +# (c) 2014, Kevin Carter +# +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import base64 +import os +import stat +import sys + +import memcache +try: + from Crypto.Cipher import AES + from Crypto import Random + + ENCRYPT_IMPORT = True +except ImportError: + ENCRYPT_IMPORT = False + +# import module snippets +from ansible.module_utils.basic import * + +DOCUMENTATION = """ +--- +module: memcached +version_added: "1.6.6" +short_description: + - Add, remove, and get items from memcached +description: + - Add, remove, and get items from memcached +options: + name: + description: + - Memcached key name + required: true + content: + description: + - Add content to memcached. Only used when state is 'present'. + required: false + file_path: + description: + - This can be used with state 'present' and 'retrieve'. When set + with state 'present' the contents of a file will be used, when + set with state 'retrieve' the contents of the memcached key will + be written to a file. + required: false + state: + description: + - ['absent', 'present', 'retrieve'] + required: true + server: + description: + - server IP address and port. This can be a comma separated list of + servers to connect to. + required: true + encrypt_string: + description: + - Encrypt/Decrypt a memcached object using a provided value. + required: false + dir_mode: + description: + - If a directory is created when using the ``file_path`` argument + the directory will be created with a set mode. + default: '0755' + required: false + file_mode: + description: + - If a file is created when using the ``file_path`` argument + the file will be created with a set mode. + default: '0644' + required: false + expires: + description: + - Seconds until an item is expired from memcached. + default: 300 + required: false +notes: + - The "absent" state will remove an item from memcached. + - The "present" state will place an item from a string or a file into + memcached. + - The "retrieve" state will get an item from memcached and return it as a + string. If a ``file_path`` is set this module will also write the value + to a file. + - All items added into memcached are base64 encoded. + - All items retrieved will attempt base64 decode and return the string + value if not applicable. + - Items retrieve from memcached are returned within a "value" key unless + a ``file_path`` is specified which would then write the contents of the + memcached key to a file. + - The ``file_path`` and ``content`` fields are mutually exclusive. + - If you'd like to encrypt items in memcached PyCrypto is a required. +requirements: + - "python-memcached" +optional_requirements: + - "pycrypto" +author: Kevin Carter +""" + +EXAMPLES = """ +# Add an item into memcached. +- memcached: + name: "key_name" + content: "Super awesome value" + state: "present" + server: "localhost:11211" + +# Read the contents of a memcached key, returned as "memcached_phrase.value". +- memcached: + name: "key_name" + state: "retrieve" + server: "localhost:11211" + register: memcached_key + +# Add the contents of a file into memcached. +- memcached: + name: "key_name" + file_path: "/home/user_name/file.txt" + state: "present" + server: "localhost:11211" + +# Write the contents of a memcached key to a file and is returned as +# "memcached_phrase.value". +- memcached: + name: "key_name" + file_path: "/home/user_name/file.txt" + state: "retrieve" + server: "localhost:11211" + register: memcached_key + +# Delete an item from memcached. +- memcached: + name: "key_name" + state: "absent" + server: "localhost:11211" +""" + +SERVER_MAX_VALUE_LENGTH = 1024 * 256 + +MAX_MEMCACHED_CHUNKS = 256 + + +class AESCipher(object): + """Encrypt an a string in using AES. + + Solution derived from "http://stackoverflow.com/a/21928790" + """ + def __init__(self, key): + if ENCRYPT_IMPORT is False: + raise ImportError( + 'PyCrypto failed to be imported. Encryption is not supported' + ' on this system until PyCrypto is installed.' + ) + + self.bs = 32 + if len(key) >= 32: + self.key = key[:32] + else: + self.key = self._pad(key) + + def encrypt(self, raw): + """Encrypt raw message. + + :param raw: ``str`` + :returns: ``str`` Base64 encoded string. + """ + raw = self._pad(raw) + iv = Random.new().read(AES.block_size) + cipher = AES.new(self.key, AES.MODE_CBC, iv) + return base64.b64encode(iv + cipher.encrypt(raw)) + + def decrypt(self, enc): + """Decrypt an encrypted message. + + :param enc: ``str`` + :returns: ``str`` + """ + enc = base64.b64decode(enc) + iv = enc[:AES.block_size] + cipher = AES.new(self.key, AES.MODE_CBC, iv) + return self._unpad(cipher.decrypt(enc[AES.block_size:])) + + def _pad(self, string): + """Pad an AES encryption key. + + :param string: ``str`` + """ + base = (self.bs - len(string) % self.bs) + back = chr(self.bs - len(string) % self.bs) + return string + base * back + + @staticmethod + def _unpad(string): + """Un-pad an AES encryption key. + + :param string: ``str`` + """ + ordinal_range = ord(string[len(string) - 1:]) + return string[:-ordinal_range] + + +class Memcached(object): + """Manage objects within memcached.""" + def __init__(self, module): + self.module = module + self.state_change = False + self.mc = None + + def router(self): + """Route all commands to their respected functions. + + If an exception happens a failure will be raised. + """ + + try: + action = getattr(self, self.module.params['state']) + self.mc = memcache.Client( + self.module.params['server'].split(','), + server_max_value_length=SERVER_MAX_VALUE_LENGTH, + debug=0 + ) + facts = action() + except Exception as exp: + self._failure(error=str(exp), rc=1, msg='general exception') + else: + self.mc.disconnect_all() + self.module.exit_json( + changed=self.state_change, **facts + ) + + def _failure(self, error, rc, msg): + """Return a Failure when running an Ansible command. + + :param error: ``str`` Error that occurred. + :param rc: ``int`` Return code while executing an Ansible command. + :param msg: ``str`` Message to report. + """ + + self.module.fail_json(msg=msg, rc=rc, err=error) + + def absent(self): + """Remove a key from memcached. + + If the value is not deleted when instructed to do so an exception will + be raised. + + :return: ``dict`` + """ + + key_name = self.module.params['name'] + get_keys = [ + '%s.%s' % (key_name, i) for i in xrange(MAX_MEMCACHED_CHUNKS) + ] + self.mc.delete_multi(get_keys) + value = self.mc.get_multi(get_keys) + if not value: + self.state_change = True + return {'absent': True, 'key': self.module.params['name']} + else: + self._failure( + error='Memcache key not deleted', + rc=1, + msg='Failed to remove an item from memcached please check your' + ' memcached server for issues. If you are load balancing' + ' memcached, attempt to connect to a single node.' + ) + + @staticmethod + def _decode_value(value): + """Return a ``str`` from a base64 decoded value. + + If the content is not a base64 ``str`` the raw value will be returned. + + :param value: ``str`` + :return: + """ + + try: + b64_value = base64.decodestring(value) + except Exception: + return value + else: + return b64_value + + def _encode_value(self, value): + """Return a base64 encoded value. + + If the value can't be base64 encoded an excption will be raised. + + :param value: ``str`` + :return: ``str`` + """ + + try: + b64_value = base64.encodestring(value) + except Exception as exp: + self._failure( + error=str(exp), + rc=1, + msg='The value provided can not be Base64 encoded.' + ) + else: + return b64_value + + def _file_read(self, full_path, pass_on_error=False): + """Read the contents of a file. + + This will read the contents of a file. If the ``full_path`` does not + exist an exception will be raised. + + :param full_path: ``str`` + :return: ``str`` + """ + + try: + with open(full_path, 'rb') as f: + o_value = f.read() + except IOError as exp: + if pass_on_error is False: + self._failure( + error=str(exp), + rc=1, + msg="The file you've specified does not exist. Please" + " check your full path @ [ %s ]." % full_path + ) + else: + return None + else: + return o_value + + def _chown(self, path, mode_type): + """Chown a file or directory based on a given mode type. + + If the file is modified the state will be changed. + + :param path: ``str`` + :param mode_type: ``str`` + """ + mode = self.module.params.get(mode_type) + # Ensure that the mode type is a string. + mode = str(mode) + _mode = oct(stat.S_IMODE(os.stat(path).st_mode)) + if _mode != mode or _mode[1:] != mode: + os.chmod(path, int(mode, 8)) + self.state_change = True + + def _file_write(self, full_path, value): + """Write the contents of ``value`` to the ``full_path``. + + This will return True upon success and will raise an exception upon + failure. + + :param full_path: ``str`` + :param value: ``str`` + :return: ``bol`` + """ + + try: + # Ensure that the directory exists + dir_path = os.path.dirname(full_path) + try: + os.makedirs(dir_path) + except OSError as exp: + if exp.errno == errno.EEXIST and os.path.isdir(dir_path): + pass + else: + self._failure( + error=str(exp), + rc=1, + msg="The directory [ %s ] does not exist and couldn't" + " be created. Please check the path and that you" + " have permission to write the file." + ) + + # Ensure proper directory permissions + self._chown(path=dir_path, mode_type='dir_mode') + + # Write contents of a cached key to a file. + with open(full_path, 'wb') as f: + if isinstance(value, list): + f.writelines(value) + else: + f.write(value) + + # Ensure proper file permissions + self._chown(path=full_path, mode_type='file_mode') + + except IOError as exp: + self._failure( + error=str(exp), + rc=1, + msg="There was an issue while attempting to write to the" + " file [ %s ]. Please check your full path and" + " permissions." % full_path + ) + else: + return True + + def retrieve(self): + """Return a value from memcached. + + If ``file_path`` is specified the value of the memcached key will be + written to a file at the ``file_path`` location. If the value of a key + is None, an exception will be raised. + + :returns: ``dict`` + """ + + key_name = self.module.params['name'] + get_keys = [ + '%s.%s' % (key_name, i) for i in xrange(MAX_MEMCACHED_CHUNKS) + ] + multi_value = self.mc.get_multi(get_keys) + if multi_value: + value = ''.join([i for i in multi_value.values() if i is not None]) + # Get the file path if specified. + file_path = self.module.params.get('file_path') + if file_path is not None: + full_path = os.path.abspath(os.path.expanduser(file_path)) + + # Decode cached value + encrypt_string = self.module.params.get('encrypt_string') + if encrypt_string: + _d_value = AESCipher(key=encrypt_string) + d_value = _d_value.decrypt(enc=value) + if not d_value: + d_value = self._decode_value(value=value) + else: + d_value = self._decode_value(value=value) + + o_value = self._file_read( + full_path=full_path, pass_on_error=True + ) + + # compare old value to new value and write if different + if o_value != d_value: + self.state_change = True + self._file_write(full_path=full_path, value=d_value) + + return { + 'present': True, + 'key': self.module.params['name'], + 'value': value, + 'file_path': full_path + } + else: + return { + 'present': True, + 'key': self.module.params['name'], + 'value': value + } + else: + self._failure( + error='Memcache key not found', + rc=1, + msg='The key you specified was not found within memcached. ' + 'If you are load balancing memcached, attempt to connect' + ' to a single node.' + ) + + def present(self): + """Create and or update a key within Memcached. + + The state processed here is present. This state will ensure that + content is written to a memcached server. When ``file_path`` is + specified the content will be read in from a file. + """ + + file_path = self.module.params.get('file_path') + if file_path is not None: + full_path = os.path.abspath(os.path.expanduser(file_path)) + # Read the contents of a file into memcached. + o_value = self._file_read(full_path=full_path) + else: + o_value = self.module.params['content'] + + # Encode cached value + encrypt_string = self.module.params.get('encrypt_string') + if encrypt_string: + _d_value = AESCipher(key=encrypt_string) + d_value = _d_value.encrypt(raw=o_value) + else: + d_value = self._encode_value(value=o_value) + + compare = 1024 * 128 + chunks = sys.getsizeof(d_value) / compare + if chunks == 0: + chunks = 1 + elif chunks > MAX_MEMCACHED_CHUNKS: + self._failure( + error='Memcache content too large', + rc=1, + msg='The content that you are attempting to cache is larger' + ' than [ %s ] megabytes.' + % ((compare * MAX_MEMCACHED_CHUNKS / 1024 / 1024)) + ) + + step = len(d_value) / chunks + if step == 0: + step = 1 + + key_name = self.module.params['name'] + split_d_value = {} + count = 0 + for i in range(0, len(d_value), step): + split_d_value['%s.%s' % (key_name, count)] = d_value[i:i + step] + count += 1 + + value = self.mc.set_multi( + mapping=split_d_value, + time=self.module.params['expires'], + min_compress_len=2048 + ) + + if not value: + self.state_change = True + return { + 'present': True, + 'key': self.module.params['name'] + } + else: + self._failure( + error='Memcache content not created', + rc=1, + msg='The content you attempted to place within memcached' + ' was not created. If you are load balancing' + ' memcached, attempt to connect to a single node.' + ' Returned a value of unstored keys [ %s ] - Original' + ' Connection [ %s ]' + % (value, [i.__dict__ for i in self.mc.servers]) + ) + + +def main(): + """Main ansible run method.""" + module = AnsibleModule( + argument_spec=dict( + name=dict( + type='str', + required=True + ), + content=dict( + type='str', + required=False + ), + file_path=dict( + type='str', + required=False + ), + state=dict( + type='str', + required=True + ), + server=dict( + type='str', + required=True + ), + expires=dict( + type='int', + default=300, + required=False + ), + file_mode=dict( + type='str', + default='0644', + required=False + ), + dir_mode=dict( + type='str', + default='0755', + required=False + ), + encrypt_string=dict( + type='str', + required=False + ) + ), + supports_check_mode=False, + mutually_exclusive=[ + ['content', 'file_path'] + ] + ) + ms = Memcached(module=module) + ms.router() + +if __name__ == '__main__': + main() diff --git a/library/name2int b/library/name2int new file mode 100644 index 0000000..5759adc --- /dev/null +++ b/library/name2int @@ -0,0 +1,79 @@ +#!/usr/bin/python +# (c) 2014, Kevin Carter +# +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import hashlib +import platform + +# import module snippets +from ansible.module_utils.basic import * + +DOCUMENTATION = """ +--- +module: name2int +version_added: "1.6.6" +short_description: + - hash a host name and return an integer +description: + - hash a host name and return an integer +options: + name: + description: + - login username + required: true +author: Kevin Carter +""" + +EXAMPLES = """ +# Create a new container +- name2int: + name: "Some-hostname.com" +""" + + +class HashHostname(object): + def __init__(self, module): + """Generate an integer from a name.""" + self.module = module + + def return_hashed_host(self, name): + hashed_name = hashlib.md5(name).hexdigest() + hash_int = int(hashed_name, 32) + real_int = int(hash_int % 300) + return real_int + + +def main(): + module = AnsibleModule( + argument_spec=dict( + name=dict( + required=True + ) + ), + supports_check_mode=False + ) + try: + sm = HashHostname(module=module) + int_value = sm.return_hashed_host(platform.node()) + resp = {'int_value': int_value} + module.exit_json(changed=True, **resp) + except Exception as exp: + resp = {'stderr': exp} + module.fail_json(msg='Failed Process', **resp) + +if __name__ == '__main__': + main() diff --git a/library/neutron b/library/neutron new file mode 100644 index 0000000..c5dbc6d --- /dev/null +++ b/library/neutron @@ -0,0 +1,422 @@ +#!/usr/bin/env python +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import keystoneclient.v3.client as ksclient +from neutronclient.neutron import client as nclient + +# import module snippets +from ansible.module_utils.basic import * + + +DOCUMENTATION = """ +--- +module: neutron +short_description: + - Basic module for interacting with openstack neutron +description: + - Basic module for interacting with openstack neutron +options: + command: + description: + - Operation for the module to perform. Currently available + choices: + - create_network + - create_subnet + - create_router + - add_router_interface + required: True + openrc_path: + decription: + - Path to openrc file from which credentials and keystone endpoint + will be extracted + net_name: + description: + - Name of network + subnet_name: + description: + - Name of subnet + router_name: + description: + - Name of router + cidr: + description: + - Specify CIDR to use when creating subnet + provider_physical_network: + description: + - Specify provider:physical_network when creating network + provider_network_type: + description: + - Specify provider:network_type when creating network + provider_segmentation_id: + description: + - Specify provider:segmentation_id when creating network + router_external: + description: + - Specify router:external' when creating network + external_gateway_info: + description: + - Specify external_gateway_info when creating router + insecure: + description: + - Explicitly allow client to perform "insecure" TLS + choices: + - false + - true + default: false +author: Hugh Saunders +""" + +EXAMPLES = """ +- name: Create private network + neutron: + command: create_network + openrc_path: /root/openrc + net_name: private +- name: Create public network + neutron: + command: create_network + openrc_path: /root/openrc + net_name: public + provider_network_type: flat + provider_physical_network: vlan + router_external: true +- name: Create private subnet + neutron: + command: create_subnet + openrc_path: /root/openrc + net_name: private + subnet_name: private-subnet + cidr: "192.168.74.0/24" +- name: Create public subnet + neutron: + command: create_subnet + openrc_path: /root/openrc + net_name: public + subnet_name: public-subnet + cidr: "10.1.13.0/24" +- name: Create router + neutron: + command: create_router + openrc_path: /root/openrc + router_name: router + external_gateway_info: public +- name: Add private subnet to router + neutron: + command: add_router_interface + openrc_path: /root/openrc + router_name: router + subnet_name: private-subnet +""" + + +COMMAND_MAP = { + 'create_network': { + 'variables': [ + 'net_name', + 'provider_physical_network', + 'provider_network_type', + 'provider_segmentation_id', + 'router_external', + 'tenant_id' + ] + }, + 'create_subnet': { + 'variables': [ + 'net_name', + 'subnet_name', + 'cidr', + 'tenant_id' + ] + }, + 'create_router': { + 'variables': [ + 'router_name', + 'external_gateway_info', + 'tenant_id' + ] + }, + 'add_router_interface': { + 'variables': [ + 'router_name', + 'subnet_name' + ] + } +} + + +class ManageNeutron(object): + def __init__(self, module): + self.state_change = False + self.neutron = None + self.keystone = None + self.module = module + + def command_router(self): + """Run the command as its provided to the module.""" + command_name = self.module.params['command'] + if command_name not in COMMAND_MAP: + self.failure( + error='No Command Found', + rc=2, + msg='Command [ %s ] was not found.' % command_name + ) + + action_command = COMMAND_MAP[command_name] + if hasattr(self, '_%s' % command_name): + action = getattr(self, '_%s' % command_name) + try: + self._keystone_authenticate() + self._init_neutron() + except Exception as e: + self.module.fail_json( + err="Initialisation Error: %s" % e, + rc=2, msg=str(e)) + facts = action(variables=action_command['variables']) + if facts is None: + self.module.exit_json(changed=self.state_change) + else: + self.module.exit_json( + changed=self.state_change, + ansible_facts=facts + ) + else: + self.failure( + error='Command not in ManageNeutron class', + rc=2, + msg='Method [ %s ] was not found.' % command_name + ) + + @staticmethod + def _facts(resource_type, resource_data): + """Return a dict for our Ansible facts.""" + key = 'neutron_%s' % resource_type + facts = {key: {}} + for f in resource_data[resource_type]: + res_name = f['name'] + del f['name'] + facts[key][res_name] = f + + return facts + + def _get_vars(self, variables, required=None): + """Return a dict of all variables as found within the module. + + :param variables: ``list`` List of all variables that are available to + use within the Neutron Command. + :param required: ``list`` Name of variables that are required. + """ + return_dict = {} + for variable in variables: + return_dict[variable] = self.module.params.get(variable) + else: + if isinstance(required, list): + for var_name in required: + check = return_dict.get(var_name) + if check is None: + self.failure( + error='Missing [ %s ] from Task or found a None' + ' value' % var_name, + rc=000, + msg='variables %s - available params [ %s ]' + % (variables, self.module.params) + ) + return return_dict + + def failure(self, error, rc, msg): + """Return a Failure when running an Ansible command. + + :param error: ``str`` Error that occurred. + :param rc: ``int`` Return code while executing an Ansible command. + :param msg: ``str`` Message to report. + """ + self.module.fail_json(msg=msg, rc=rc, err=error) + + def _parse_openrc(self): + """Get credentials from an openrc file.""" + openrc_path = self.module.params['openrc_path'] + line_re = re.compile('^export (?POS_\w*)=(?P[^\n]*)') + with open(openrc_path) as openrc: + matches = [line_re.match(l) for l in openrc] + return dict( + (g.groupdict()['key'], g.groupdict()['value']) + for g in matches if g + ) + + def _keystone_authenticate(self): + """Authenticate with Keystone.""" + openrc = self._parse_openrc() + insecure = self.module.params['insecure'] + self.keystone = ksclient.Client(insecure=insecure, + username=openrc['OS_USERNAME'], + password=openrc['OS_PASSWORD'], + project_name=openrc['OS_PROJECT_NAME'], + auth_url=openrc['OS_AUTH_URL']) + + def _init_neutron(self): + """Create neutron client object using token and url from keystone.""" + openrc = self._parse_openrc() + self.neutron = nclient.Client( + '2.0', + endpoint_url=self.keystone.service_catalog.url_for( + service_type='network', + endpoint_type=openrc['OS_ENDPOINT_TYPE']), + token=self.keystone.get_token(self.keystone.session)) + + def _get_resource_by_name(self, resource_type, resource_name): + action = getattr(self.neutron, 'list_%s' % resource_type) + resource = action(name=resource_name)[resource_type] + + if resource: + return resource[0]['id'] + else: + return None + + def _create_network(self, variables): + required_vars = ['net_name'] + variables_dict = self._get_vars(variables, required=required_vars) + net_name = variables_dict.pop('net_name') + provider_physical_network = variables_dict.pop( + 'provider_physical_network' + ) + provider_network_type = variables_dict.pop('provider_network_type') + provider_segmentation_id = variables_dict.pop( + 'provider_segmentation_id' + ) + router_external = variables_dict.pop('router_external') + tenant_id = variables_dict.pop('tenant_id') + + if not self._get_resource_by_name('networks', net_name): + n = {"name": net_name, "admin_state_up": True} + if provider_physical_network: + n['provider:physical_network'] = provider_physical_network + if provider_network_type: + n['provider:network_type'] = provider_network_type + if provider_segmentation_id: + n['provider:segmentation_id'] = str(provider_segmentation_id) + if router_external: + n['router:external'] = router_external + if tenant_id: + n['tenant_id'] = tenant_id + + self.state_change = True + self.neutron.create_network({"network": n}) + + return self._facts('networks', self.neutron.list_networks()) + + def _create_subnet(self, variables): + required_vars = ['net_name', 'subnet_name', 'cidr'] + variables_dict = self._get_vars(variables, required=required_vars) + net_name = variables_dict.pop('net_name') + subnet_name = variables_dict.pop('subnet_name') + cidr = variables_dict.pop('cidr') + network_id = self._get_resource_by_name('networks', net_name) + tenant_id = variables_dict.pop('tenant_id') + + if not network_id: + self.failure( + error='Network not found', + rc=1, + msg='The specified network could not be found' + ) + if not self.neutron.list_subnets(cidr=cidr, + network_id=network_id)['subnets']: + self.state_change = True + s = {"name": subnet_name, "cidr": cidr, "ip_version": 4, + "network_id": network_id} + if tenant_id: + s["tenant_id"] = tenant_id + self.neutron.create_subnet({"subnet": s}) + return self._facts('subnets', self.neutron.list_subnets()) + + def _create_router(self, variables): + required_vars = ['router_name', 'external_gateway_info'] + variables_dict = self._get_vars(variables, required=required_vars) + router_name = variables_dict.pop('router_name') + external_gateway_info = variables_dict.pop('external_gateway_info') + tenant_id = variables_dict.pop('tenant_id') + + if not self._get_resource_by_name('routers', router_name): + self.state_change = True + r = {'name': router_name} + if external_gateway_info: + network_id = self._get_resource_by_name('networks', + external_gateway_info) + r['external_gateway_info'] = {'network_id': network_id} + if tenant_id: + r['tenant_id'] = tenant_id + self.neutron.create_router({'router': r}) + + return self._facts('routers', self.neutron.list_routers()) + + def _add_router_interface(self, variables): + required_vars = ['router_name', 'subnet_name'] + variables_dict = self._get_vars(variables, required=required_vars) + router_name = variables_dict.pop('router_name') + subnet_name = variables_dict.pop('subnet_name') + router_id = self._get_resource_by_name('routers', router_name) + subnet_id = self._get_resource_by_name('subnets', subnet_name) + + if not router_id: + self.failure( + error='Router not found', + rc=1, + msg='The specified router could not be found' + ) + + if not subnet_id: + self.failure( + error='Subnet not found', + rc=1, + msg='The specified subnet could not be found' + ) + + found = False + + for port in self.neutron.list_ports(device_id=router_id)['ports']: + for fixed_ips in port['fixed_ips']: + if fixed_ips['subnet_id'] == subnet_id: + found = True + if not found: + self.state_change = True + self.neutron.add_interface_router(router_id, + {'subnet_id': subnet_id}) + + +def main(): + module = AnsibleModule( + argument_spec=dict( + command=dict(required=True, choices=COMMAND_MAP.keys()), + openrc_path=dict(required=True), + net_name=dict(required=False), + subnet_name=dict(required=False), + cidr=dict(required=False), + provider_physical_network=dict(required=False), + provider_network_type=dict(required=False), + provider_segmentation_id=dict(required=False), + router_external=dict(required=False), + router_name=dict(required=False), + external_gateway_info=dict(required=False), + tenant_id=dict(required=False), + insecure=dict(default=False, required=False, + choices=BOOLEANS + ['True', 'False']) + ), + supports_check_mode=False + ) + mn = ManageNeutron(module) + mn.command_router() + +if __name__ == '__main__': + main() diff --git a/library/provider_networks b/library/provider_networks new file mode 100644 index 0000000..d3aefa7 --- /dev/null +++ b/library/provider_networks @@ -0,0 +1,283 @@ +#!/usr/bin/python +# (c) 2014, Kevin Carter +# +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# import module snippets +from ansible.module_utils.basic import * + + +DOCUMENTATION = """ +--- +module: provider_networks +version_added: "1.8.4" +short_description: + - Parse a list of networks and return data that Ansible can use +description: + - Parse a list of networks and return data that Ansible can use +options: + provider_networks: + description: + - List of networks to parse + required: true + is_metal: + description: + - Enable handling of on metal hosts + required: false + bind_prefix: + description: + - Add a prefix to all network interfaces. + required: false +author: Kevin Carter +""" + +EXAMPLES = """ +## This is what the provider_networks list should look like. +# provider_networks: +# - network: +# container_bridge: "br-mgmt" +# container_type: "veth" +# container_interface: "eth1" +# ip_from_q: "container" +# type: "raw" +# group_binds: +# - all_containers +# - hosts +# is_container_address: true +# is_ssh_address: true +# - network: +# container_bridge: "br-vxlan" +# container_type: "veth" +# container_interface: "eth10" +# ip_from_q: "tunnel" +# type: "vxlan" +# range: "1:1000" +# net_name: "vxlan" +# group_binds: +# - neutron_linuxbridge_agent +# - network: +# container_bridge: "br-vlan" +# container_type: "veth" +# container_interface: "eth12" +# host_bind_override: "eth12" +# type: "flat" +# net_name: "flat" +# group_binds: +# - neutron_linuxbridge_agent +# - network: +# container_bridge: "br-vlan" +# container_type: "veth" +# container_interface: "eth11" +# host_bind_override: "eth11" +# type: "vlan" +# range: "1:1, 101:101" +# net_name: "vlan" +# group_binds: +# - neutron_linuxbridge_agent +# - network: +# container_bridge: "br-storage" +# container_type: "veth" +# container_interface: "eth2" +# ip_from_q: "storage" +# type: "raw" +# group_binds: +# - glance_api +# - cinder_api +# - cinder_volume +# - nova_compute +# - swift_proxy + + +- name: Test provider networks + provider_networks: + provider_networks: "{{ provider_networks }}" + register: pndata1 + +- name: Test provider networks is metal + provider_networks: + provider_networks: "{{ provider_networks }}" + is_metal: true + register: pndata2 + +- name: Test provider networks with prfix + provider_networks: + provider_networks: "{{ provider_networks }}" + bind_prefix: "brx" + is_metal: true + register: pndata3 + +## Module output: +# { +# "network_flat_networks": "flat", +# "network_flat_networks_list": [ +# "flat" +# ], +# "network_mappings": "flat:brx-eth12,vlan:brx-eth11", +# "network_mappings_list": [ +# "flat:brx-eth12", +# "vlan:brx-eth11" +# ], +# "network_types": "vxlan,flat,vlan", +# "network_types_list": [ +# "vxlan", +# "flat", +# "vlan" +# ], +# "network_vlan_ranges": "vlan:1:1,vlan:1024:1025", +# "network_vlan_ranges_list": [ +# "vlan:1:1", +# "vlan:1024:1025" +# ], +# "network_vxlan_ranges": "1:1000", +# "network_vxlan_ranges_list": [ +# "1:1000" +# ] +# } +""" + + +class ProviderNetworksParsing(object): + def __init__(self, module): + """Generate an integer from a name. + + :param module: Load the ansible module + :type module: ``object`` + """ + self.module = module + self.network_vlan_ranges = list() + self.network_vxlan_ranges = list() + self.network_flat_networks = list() + self.network_mappings = list() + self.network_types = list() + + def load_networks(self, provider_networks, is_metal=False, + bind_prefix=None): + """Load the lists of network and network data types. + + :param provider_networks: list of networks defined in user_config + :type provider_networks: ``list`` + :param is_metal: Enable of disable handling of on metal nodes + :type is_metal: ``bol`` + :param bind_prefix: Pre-interface prefix forced within the network map + :type bind_prefix: ``str`` + """ + + for net in provider_networks: + if net['network']['type'] == "vlan": + if "vlan" not in self.network_types: + self.network_types.append('vlan') + for vlan_range in net['network']['range'].split(','): + self.network_vlan_ranges.append( + '%s:%s' % ( + net['network']['net_name'], vlan_range.strip() + ) + ) + elif net['network']['type'] == "vxlan": + if "vxlan" not in self.network_types: + self.network_types.append('vxlan') + self.network_vxlan_ranges.append(net['network']['range']) + elif net['network']['type'] == "flat": + if "flat" not in self.network_types: + self.network_types.append('flat') + self.network_flat_networks.append( + net['network']['net_name'] + ) + + # Create the network mappings + if net['network']['type'] not in ['raw', 'vxlan']: + if 'net_name' in net['network']: + if is_metal: + if 'host_bind_override' in net['network']: + bind_device = net['network']['host_bind_override'] + else: + bind_device = net['network']['container_bridge'] + else: + bind_device = net['network']['container_interface'] + + if bind_prefix: + bind_device = '%s-%s' % (bind_prefix, bind_device) + + self.network_mappings.append( + '%s:%s' % ( + net['network']['net_name'], + bind_device + ) + ) + + +def main(): + + # Add in python True False + BOOLEANS.extend(['False', 'True']) + BOOLEANS_TRUE.append('True') + BOOLEANS_FALSE.append('False') + + module = AnsibleModule( + argument_spec=dict( + provider_networks=dict( + type='list', + required=True + ), + is_metal=dict( + choices=BOOLEANS, + default='false' + ), + bind_prefix=dict( + type='str', + required=False, + default=None + ) + ), + supports_check_mode=False + ) + + try: + is_metal = module.params.get('is_metal') + if is_metal in BOOLEANS_TRUE: + module.params['is_metal'] = True + else: + module.params['is_metal'] = False + + pnp = ProviderNetworksParsing(module=module) + pnp.load_networks( + provider_networks=module.params.get('provider_networks'), + is_metal=module.params.get('is_metal'), + bind_prefix=module.params.get('bind_prefix') + ) + + # Response dictionary, this adds commas to all list items in string + # format as well as preserves the list functionality for future data + # processing. + resp = { + 'network_vlan_ranges': ','.join(pnp.network_vlan_ranges), + 'network_vlan_ranges_list': pnp.network_vlan_ranges, + 'network_vxlan_ranges': ','.join(pnp.network_vxlan_ranges), + 'network_vxlan_ranges_list': pnp.network_vxlan_ranges, + 'network_flat_networks': ','.join(pnp.network_flat_networks), + 'network_flat_networks_list': pnp.network_flat_networks, + 'network_mappings': ','.join(pnp.network_mappings), + 'network_mappings_list': pnp.network_mappings, + 'network_types': ','.join(pnp.network_types), + 'network_types_list': pnp.network_types + } + + module.exit_json(changed=True, **resp) + except Exception as exp: + resp = {'stderr': exp} + module.fail_json(msg='Failed Process', **resp) + +if __name__ == '__main__': + main() diff --git a/lookups/py_pkgs.py b/lookups/py_pkgs.py new file mode 100644 index 0000000..a7af487 --- /dev/null +++ b/lookups/py_pkgs.py @@ -0,0 +1,615 @@ +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# (c) 2014, Kevin Carter + +import os +import re +import traceback + +from distutils.version import LooseVersion +from ansible import __version__ as __ansible_version__ +import yaml + + +# Used to keep track of git package parts as various files are processed +GIT_PACKAGE_DEFAULT_PARTS = dict() + + +ROLE_PACKAGES = dict() + + +REQUIREMENTS_FILE_TYPES = [ + 'global-requirements.txt', + 'test-requirements.txt', + 'dev-requirements.txt', + 'requirements.txt', + 'global-requirement-pins.txt' +] + + +# List of variable names that could be used within the yaml files that +# represent lists of python packages. +BUILT_IN_PIP_PACKAGE_VARS = [ + 'service_pip_dependencies', + 'pip_common_packages', + 'pip_container_packages', + 'pip_packages' +] + + +PACKAGE_MAPPING = { + 'packages': set(), + 'remote_packages': set(), + 'remote_package_parts': list(), + 'role_packages': dict() +} + + +def map_base_and_remote_packages(package, package_map): + """Determine whether a package is a base package or a remote package + and add to the appropriate set. + + :type package: ``str`` + :type package_map: ``dict`` + """ + if package.startswith(('http:', 'https:', 'git+')): + if '@' not in package: + package_map['packages'].add(package) + else: + git_parts = git_pip_link_parse(package) + package_name = git_parts[-1] + if not package_name: + package_name = git_pip_link_parse(package)[0] + + for rpkg in list(package_map['remote_packages']): + rpkg_name = git_pip_link_parse(rpkg)[-1] + if not rpkg_name: + rpkg_name = git_pip_link_parse(package)[0] + + if rpkg_name == package_name: + package_map['remote_packages'].remove(rpkg) + package_map['remote_packages'].add(package) + break + else: + package_map['remote_packages'].add(package) + else: + package_map['packages'].add(package) + + +def parse_remote_package_parts(package_map): + """Parse parts of each remote package and add them to + the remote_package_parts list. + + :type package_map: ``dict`` + """ + keys = [ + 'name', + 'version', + 'fragment', + 'url', + 'original', + 'egg_name' + ] + remote_pkg_parts = [ + dict( + zip( + keys, git_pip_link_parse(i) + ) + ) for i in package_map['remote_packages'] + ] + package_map['remote_package_parts'].extend(remote_pkg_parts) + package_map['remote_package_parts'] = list( + dict( + (i['name'], i) + for i in package_map['remote_package_parts'] + ).values() + ) + + +def map_role_packages(package_map): + """Add and sort packages belonging to a role to the role_packages dict. + + :type package_map: ``dict`` + """ + for k, v in ROLE_PACKAGES.items(): + role_pkgs = package_map['role_packages'][k] = list() + for pkg_list in v.values(): + role_pkgs.extend(pkg_list) + else: + package_map['role_packages'][k] = sorted(set(role_pkgs)) + + +def map_base_package_details(package_map): + """Parse package version and marker requirements and add to the + base packages set. + + :type package_map: ``dict`` + """ + check_pkgs = dict() + base_packages = sorted(list(package_map['packages'])) + for pkg in base_packages: + name, versions, markers = _pip_requirement_split(pkg) + if versions and markers: + versions = '%s;%s' % (versions, markers) + elif not versions and markers: + versions = ';%s' % markers + + if name in check_pkgs: + if versions and not check_pkgs[name]: + check_pkgs[name] = versions + else: + check_pkgs[name] = versions + else: + return_pkgs = list() + for k, v in check_pkgs.items(): + if v: + return_pkgs.append('%s%s' % (k, v)) + else: + return_pkgs.append(k) + package_map['packages'] = set(return_pkgs) + + +def git_pip_link_parse(repo): + """Return a tuple containing the parts of a git repository. + + Example parsing a standard git repo: + >>> git_pip_link_parse('git+https://github.com/username/repo-name@tag') + ('repo-name', + 'tag', + None, + 'https://github.com/username/repo', + 'git+https://github.com/username/repo@tag', + 'repo_name') + + Example parsing a git repo that uses an installable from a subdirectory: + >>> git_pip_link_parse( + ... 'git+https://github.com/username/repo@tag#egg=plugin.name' + ... '&subdirectory=remote_path/plugin.name' + ... ) + ('plugin.name', + 'tag', + 'remote_path/plugin.name', + 'https://github.com/username/repo', + 'git+https://github.com/username/repo@tag#egg=plugin.name&' + 'subdirectory=remote_path/plugin.name', + 'plugin.name') + + :param repo: git repo string to parse. + :type repo: ``str`` + :returns: ``tuple`` + """'meta' + + def _meta_return(meta_data, item): + """Return the value of an item in meta data.""" + + return meta_data.lstrip('#').split('%s=' % item)[-1].split('&')[0] + + _git_url = repo.split('+') + if len(_git_url) >= 2: + _git_url = _git_url[1] + else: + _git_url = _git_url[0] + + git_branch_sha = _git_url.split('@') + if len(git_branch_sha) > 2: + branch = git_branch_sha.pop() + url = '@'.join(git_branch_sha) + elif len(git_branch_sha) > 1: + url, branch = git_branch_sha + else: + url = git_branch_sha[0] + branch = 'master' + + egg_name = name = os.path.basename(url.rstrip('/')) + egg_name = egg_name.replace('-', '_') + + _branch = branch.split('#') + branch = _branch[0] + + plugin_path = None + # Determine if the package is a plugin type + if len(_branch) > 1: + if 'subdirectory=' in _branch[-1]: + plugin_path = _meta_return(_branch[-1], 'subdirectory') + name = os.path.basename(plugin_path) + + if 'egg=' in _branch[-1]: + egg_name = _meta_return(_branch[-1], 'egg') + egg_name = egg_name.replace('-', '_') + + if 'gitname=' in _branch[-1]: + name = _meta_return(_branch[-1], 'gitname') + + return name.lower(), branch, plugin_path, url, repo, egg_name + + +def _pip_requirement_split(requirement): + """Split pip versions from a given requirement. + + The method will return the package name, versions, and any markers. + + :type requirement: ``str`` + :returns: ``tuple`` + """ + version_descriptors = "(>=|<=|>|<|==|~=|!=)" + requirement = requirement.split(';') + requirement_info = re.split(r'%s\s*' % version_descriptors, requirement[0]) + name = requirement_info[0] + marker = None + if len(requirement) > 1: + marker = requirement[-1] + versions = None + if len(requirement_info) > 1: + versions = ''.join(requirement_info[1:]) + + return name, versions, marker + + +class DependencyFileProcessor(object): + def __init__(self, local_path): + """Find required files. + + :type local_path: ``str`` + :return: + """ + self.pip = dict() + self.pip['git_package'] = list() + self.pip['py_package'] = list() + self.pip['git_data'] = list() + self.git_pip_install = 'git+%s@%s' + self.file_names = self._get_files(path=local_path) + + # Process everything simply by calling the method + self._process_files() + + def _py_pkg_extend(self, packages): + for pkg in packages: + pkg_name = _pip_requirement_split(pkg)[0] + for py_pkg in self.pip['py_package']: + py_pkg_name = _pip_requirement_split(py_pkg)[0] + if pkg_name == py_pkg_name: + self.pip['py_package'].remove(py_pkg) + else: + self.pip['py_package'].extend([i.lower() for i in packages]) + + @staticmethod + def _filter_files(file_names, ext): + """Filter the files and return a sorted list. + + :type file_names: + :type ext: ``str`` or ``tuple`` + :returns: ``list`` + """ + _file_names = list() + file_name_words = ['/defaults/', '/vars/', '/user_'] + file_name_words.extend(REQUIREMENTS_FILE_TYPES) + for file_name in file_names: + if file_name.endswith(ext): + if any(i in file_name for i in file_name_words): + _file_names.append(file_name) + else: + return _file_names + + @staticmethod + def _get_files(path): + """Return a list of all files in the defaults/repo_packages directory. + + :type path: ``str`` + :returns: ``list`` + """ + paths = os.walk(os.path.abspath(path)) + files = list() + for fpath, _, afiles in paths: + for afile in afiles: + files.append(os.path.join(fpath, afile)) + else: + return files + + def _check_plugins(self, git_repo_plugins, git_data): + """Check if the git url is a plugin type. + + :type git_repo_plugins: ``dict`` + :type git_data: ``dict`` + """ + for repo_plugin in git_repo_plugins: + strip_plugin_path = repo_plugin['package'].lstrip('/') + plugin = '%s/%s' % ( + repo_plugin['path'].strip('/'), + strip_plugin_path + ) + + name = git_data['name'] = os.path.basename(strip_plugin_path) + git_data['egg_name'] = name.replace('-', '_') + package = self.git_pip_install % ( + git_data['repo'], git_data['branch'] + ) + package += '#egg=%s' % git_data['egg_name'] + package += '&subdirectory=%s' % plugin + package += '&gitname=%s' % name + if git_data['fragments']: + package += '&%s' % git_data['fragments'] + + self.pip['git_data'].append(git_data) + self.pip['git_package'].append(package) + + if name not in GIT_PACKAGE_DEFAULT_PARTS: + GIT_PACKAGE_DEFAULT_PARTS[name] = git_data.copy() + else: + GIT_PACKAGE_DEFAULT_PARTS[name].update(git_data.copy()) + + @staticmethod + def _check_defaults(git_data, name, item): + """Check if a default exists and use it if an item is undefined. + + :type git_data: ``dict`` + :type name: ``str`` + :type item: ``str`` + """ + if not git_data[item] and name in GIT_PACKAGE_DEFAULT_PARTS: + check_item = GIT_PACKAGE_DEFAULT_PARTS[name].get(item) + if check_item: + git_data[item] = check_item + + def _process_git(self, loaded_yaml, git_item): + """Process git repos. + + :type loaded_yaml: ``dict`` + :type git_item: ``str`` + """ + git_data = dict() + if git_item.split('_')[0] == 'git': + prefix = '' + else: + prefix = '%s_' % git_item.split('_git_repo')[0].replace('.', '_') + + # Set the various variable definitions + repo_var = prefix + 'git_repo' + name_var = prefix + 'git_package_name' + branch_var = prefix + 'git_install_branch' + fragment_var = prefix + 'git_install_fragments' + plugins_var = prefix + 'repo_plugins' + + # get the repo definition + git_data['repo'] = loaded_yaml.get(repo_var) + + # get the repo name definition + name = git_data['name'] = loaded_yaml.get(name_var) + if not name: + name = git_data['name'] = os.path.basename( + git_data['repo'].rstrip('/') + ) + git_data['egg_name'] = name.replace('-', '_') + + # get the repo branch definition + git_data['branch'] = loaded_yaml.get(branch_var) + self._check_defaults(git_data, name, 'branch') + if not git_data['branch']: + git_data['branch'] = 'master' + + package = self.git_pip_install % (git_data['repo'], git_data['branch']) + + # get the repo fragment definitions, if any + git_data['fragments'] = loaded_yaml.get(fragment_var) + self._check_defaults(git_data, name, 'fragments') + + package += '#egg=%s' % git_data['egg_name'] + package += '&gitname=%s' % name + if git_data['fragments']: + package += '&%s' % git_data['fragments'] + + self.pip['git_package'].append(package) + self.pip['git_data'].append(git_data.copy()) + + # Set the default package parts to track data during the run + if name not in GIT_PACKAGE_DEFAULT_PARTS: + GIT_PACKAGE_DEFAULT_PARTS[name] = git_data.copy() + else: + GIT_PACKAGE_DEFAULT_PARTS[name].update() + + # get the repo plugin definitions, if any + git_data['plugins'] = loaded_yaml.get(plugins_var) + self._check_defaults(git_data, name, 'plugins') + if git_data['plugins']: + self._check_plugins( + git_repo_plugins=git_data['plugins'], + git_data=git_data + ) + + def _process_files(self): + """Process files.""" + + role_name = None + for file_name in self._filter_files(self.file_names, ('yaml', 'yml')): + with open(file_name, 'r') as f: + # If there is an exception loading the file continue + # and if the loaded_config is None continue. This makes + # no bad config gets passed to the rest of the process. + try: + loaded_config = yaml.safe_load(f.read()) + except Exception: # Broad exception so everything is caught + continue + else: + if not loaded_config: + continue + + if 'roles' in file_name: + _role_name = file_name.split('roles%s' % os.sep)[-1] + role_name = _role_name.split(os.sep)[0] + + for key, values in loaded_config.items(): + # This conditional is set to ensure we're not processes git + # repos from the defaults file which may conflict with what is + # being set in the repo_packages files. + if '/defaults/main' not in file_name: + if key.endswith('git_repo'): + self._process_git( + loaded_yaml=loaded_config, + git_item=key + ) + + if [i for i in BUILT_IN_PIP_PACKAGE_VARS if i in key]: + self._py_pkg_extend(values) + if role_name: + if role_name in ROLE_PACKAGES: + role_pkgs = ROLE_PACKAGES[role_name] + else: + role_pkgs = ROLE_PACKAGES[role_name] = dict() + + pkgs = role_pkgs.get(key, list()) + if 'optional' not in key: + pkgs.extend(values) + ROLE_PACKAGES[role_name][key] = pkgs + else: + for k, v in ROLE_PACKAGES.items(): + for item_name in v.keys(): + if key == item_name: + ROLE_PACKAGES[k][item_name].extend(values) + + for file_name in self._filter_files(self.file_names, 'txt'): + if os.path.basename(file_name) in REQUIREMENTS_FILE_TYPES: + with open(file_name, 'r') as f: + packages = [ + i.split()[0] for i in f.read().splitlines() + if i + if not i.startswith('#') + ] + self._py_pkg_extend(packages) + + +def _abs_path(path): + return os.path.abspath( + os.path.expanduser( + path + ) + ) + + +class LookupModule(object): + def __new__(class_name, *args, **kwargs): + if LooseVersion(__ansible_version__) < LooseVersion("2.0"): + from ansible import utils, errors + + class LookupModuleV1(object): + def __init__(self, basedir=None, **kwargs): + """Run the lookup module. + + :type basedir: + :type kwargs: + """ + self.basedir = basedir + + def run(self, terms, inject=None, **kwargs): + """Run the main application. + + :type terms: ``str`` + :type inject: ``str`` + :type kwargs: ``dict`` + :returns: ``list`` + """ + terms = utils.listify_lookup_plugin_terms( + terms, + self.basedir, + inject + ) + if isinstance(terms, basestring): + terms = [terms] + + return_data = PACKAGE_MAPPING + + for term in terms: + return_list = list() + try: + dfp = DependencyFileProcessor( + local_path=_abs_path(str(term)) + ) + return_list.extend(dfp.pip['py_package']) + return_list.extend(dfp.pip['git_package']) + except Exception as exp: + raise errors.AnsibleError( + 'lookup_plugin.py_pkgs(%s) returned "%s" error "%s"' % ( + term, + str(exp), + traceback.format_exc() + ) + ) + + for item in return_list: + map_base_and_remote_packages(item, return_data) + else: + parse_remote_package_parts(return_data) + else: + map_role_packages(return_data) + map_base_package_details(return_data) + # Sort everything within the returned data + for key, value in return_data.items(): + if isinstance(value, (list, set)): + return_data[key] = sorted(value) + return [return_data] + return LookupModuleV1(*args, **kwargs) + + else: + from ansible.errors import AnsibleError + from ansible.plugins.lookup import LookupBase + + class LookupModuleV2(LookupBase): + def run(self, terms, variables=None, **kwargs): + """Run the main application. + + :type terms: ``str`` + :type variables: ``str`` + :type kwargs: ``dict`` + :returns: ``list`` + """ + if isinstance(terms, basestring): + terms = [terms] + + return_data = PACKAGE_MAPPING + + for term in terms: + return_list = list() + try: + dfp = DependencyFileProcessor( + local_path=_abs_path(str(term)) + ) + return_list.extend(dfp.pip['py_package']) + return_list.extend(dfp.pip['git_package']) + except Exception as exp: + raise AnsibleError( + 'lookup_plugin.py_pkgs(%s) returned "%s" error "%s"' % ( + term, + str(exp), + traceback.format_exc() + ) + ) + + for item in return_list: + map_base_and_remote_packages(item, return_data) + else: + parse_remote_package_parts(return_data) + else: + map_role_packages(return_data) + map_base_package_details(return_data) + # Sort everything within the returned data + for key, value in return_data.items(): + if isinstance(value, (list, set)): + return_data[key] = sorted(value) + return [return_data] + return LookupModuleV2(*args, **kwargs) + +# Used for testing and debuging usage: `python plugins/lookups/py_pkgs.py ../` +if __name__ == '__main__': + import sys + import json + print(json.dumps(LookupModule().run(terms=sys.argv[1:]), indent=4)) diff --git a/meta/main.yml b/meta/main.yml new file mode 100644 index 0000000..23315d4 --- /dev/null +++ b/meta/main.yml @@ -0,0 +1,31 @@ +--- +# Copyright 2014, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +galaxy_info: + author: rcbops + description: Plugin collection + company: Rackspace + license: Apache2 + min_ansible_version: 1.6.6 + platforms: + - name: Ubuntu + versions: + - trusty + categories: + - cloud + - rabbitmq + - development + - openstack +dependencies: [] diff --git a/run_tests.sh b/run_tests.sh new file mode 100644 index 0000000..2e24671 --- /dev/null +++ b/run_tests.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Copyright 2015, Rackspace US, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euov + +ROLE_NAME=$(basename $(pwd)) +FUNCTIONAL_TEST=${FUNCTIONAL_TEST:-true} + +pushd tests + ansible-galaxy install \ + --role-file=ansible-role-requirements.yml \ + --ignore-errors \ + --force + + ansible-playbook -i inventory \ + --syntax-check \ + --list-tasks \ + -e "rolename=${ROLE_NAME}" \ + test.yml + + ansible-lint test.yml + + if ${FUNCTIONAL_TEST}; then + ansible-playbook -i inventory -e "rolename=${ROLE_NAME}" test.yml + fi +popd diff --git a/setup.cfg b/setup.cfg new file mode 100644 index 0000000..3bd1120 --- /dev/null +++ b/setup.cfg @@ -0,0 +1,24 @@ +[metadata] +name = openstack-ansible-plugins +summary = plugins for OpenStack Ansible +description-file = + README.rst +author = OpenStack +author-email = openstack-dev@lists.openstack.org +home-page = http://www.openstack.org/ +classifier = + Intended Audience :: Developers + Intended Audience :: System Administrators + License :: OSI Approved :: Apache Software License + Operating System :: POSIX :: Linux + +[build_sphinx] +all_files = 1 +build-dir = doc/build +source-dir = doc/source + +[pbr] +warnerrors = True + +[wheel] +universal = 1 diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..70c2b3f --- /dev/null +++ b/setup.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python +# Copyright (c) 2013 Hewlett-Packard Development Company, L.P. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT +import setuptools + +setuptools.setup( + setup_requires=['pbr'], + pbr=True) diff --git a/tox.ini b/tox.ini new file mode 100644 index 0000000..7ac979d --- /dev/null +++ b/tox.ini @@ -0,0 +1,87 @@ +[tox] +minversion = 1.6 +skipsdist = True +envlist = docs,pep8,bashate,ansible-syntax,ansible-lint + +[testenv] +usedevelop = True +install_command = pip install -U {opts} {packages} +setenv = VIRTUAL_ENV={envdir} +deps = -r{toxinidir}/dev-requirements.txt +commands = + /usr/bin/find . -type f -name "*.pyc" -delete + ansible-galaxy install \ + --role-file=ansible-role-requirements.yml \ + --ignore-errors \ + --force + +[testenv:docs] +commands = python setup.py build_sphinx + +# environment used by the -infra templated docs job +[testenv:venv] +deps = -r{toxinidir}/dev-requirements.txt +commands = {posargs} + +# Run hacking/flake8 check for all python files +[testenv:pep8] +deps = flake8 +whitelist_externals = bash +commands = + bash -c "grep -Irl \ + -e '!/usr/bin/env python' \ + -e '!/bin/python' \ + -e '!/usr/bin/python' \ + --exclude-dir '.*' \ + --exclude-dir 'doc' \ + --exclude-dir '*.egg' \ + --exclude-dir '*.egg-info' \ + --exclude 'tox.ini' \ + --exclude '*.sh' \ + {toxinidir} | xargs flake8 --verbose" + +[flake8] +# Ignores the following rules due to how ansible modules work in general +# F403 'from ansible.module_utils.basic import *' used; unable to detect undefined names +# H303 No wildcard (*) import. +ignore=F403,H303 + +# Run bashate check for all bash scripts +# Ignores the following rules: +# E003: Indent not multiple of 4 (we prefer to use multiples of 2) +[testenv:bashate] +deps = bashate +whitelist_externals = bash +commands = + bash -c "grep -Irl \ + -e '!/usr/bin/env bash' \ + -e '!/bin/bash' \ + -e '!/bin/sh' \ + --exclude-dir '.*' \ + --exclude-dir '*.egg' \ + --exclude-dir '*.egg-info' \ + --exclude 'tox.ini' \ + {toxinidir} | xargs bashate --verbose --ignore=E003" + +[testenv:ansible-syntax] +changedir = tests +commands = + ansible-galaxy install \ + --role-file=ansible-role-requirements.yml \ + --ignore-errors \ + --force + ansible-playbook -i inventory \ + --syntax-check \ + --list-tasks \ + -e "rolename={toxinidir}" \ + test.yml + +[testenv:ansible-lint] +changedir = tests +commands = + ansible-galaxy install \ + --role-file=ansible-role-requirements.yml \ + --ignore-errors \ + --force + ansible-lint test.yml +