Use puppet apply instead of puppet agent

At long last, the day of reckoning is here. Run puppet apply and then
copy the log files back and post them to puppetdb.

Change-Id: I919fea64df0fbb8681e91ac9425b4c43760bb3dd

doc/source/puppet.rst

@@ -5,10 +5,11 @@
 Puppet Master
 #############
 
-Puppet agent is a mechanism use to pull puppet manifests and configuration
+The puppetmaster server is named puppetmaster for historical reasons - it
-from a centralized master. This means there is only one place that needs to
+no longer runs a puppetmaster process. There is a centralized 'hiera'
-hold secure information such as passwords, and only one location for the git
+database that contains secure information such as passwords. The puppetmaster
-repo holding the modules.
+server contains all of the ansible playbooks to run puppet apply
+as well as the scripts to create new servers.
 
 At a Glance
 ===========
@@ -25,50 +26,59 @@ At a Glance
 :Resources:
   * `Puppet Language Reference <https://docs.puppetlabs.com/references/latest/type.html>`_
 
-Puppet Master
+Puppet Driving Ansible Driving Puppet
--------------
+-------------------------------------
 
-The puppet master is setup using a combination of Apache and mod passenger to
+In OpenStack Infra, there are ansible playbooks that drive the running of
-ship the data to the clients.
+``puppet apply`` on all of the hosts in the inventory. That process first
+copies appropriate ``hiera`` data files to each host, and when it is done
+it copies back the JSON report of the puppet run and submits it to
+``puppetdb``.
 
 The cron jobs, current configuration files and more can be done with ``puppet
 apply`` but first some bootstrapping needs to be done.
 
-You want to install these from puppetlabs' apt repo. There is a script in the
+You want to install these from puppetlabs' apt repo. There is a script,
-root of the system-config repository that will setup and install the
+`:file:`install_puppet.sh` in the root of the system-config repository that
-puppet client. After that you must install the puppetmaster and hiera (used to
+will setup and install the puppet client. After that you must install the
-maintain secrets on the puppet master).
+ansible playbooks and hiera config (used to maintain secrets).
 
-Puppet 3 masters can run on Trusty, Precise, and Centos 6.
+Ansible and Puppet 3 is known to run on Precise, Trusty, Centos 6 and Centos 7.
 
 .. code-block:: bash
 
    sudo su -
    git clone https://git.openstack.org/openstack-infra/system-config /opt/system-config/production
-   /opt/system-config/production/install_puppet.sh
+   bash /opt/system-config/production/install_puppet.sh
-   apt-get install puppetmaster-passenger hiera hiera-puppet
-
-Finally, install the modules, fix your hostname and use ``puppet apply`` to
-finish configuration:
-
-.. code-block:: bash
-
    bash /opt/system-config/production/install_modules.sh
    echo $REAL_HOSTNAME > /etc/hostname
    service hostname restart
    puppet apply --modulepath='/opt/system-config/production/modules:/etc/puppet/modules' -e 'include openstack_project::puppetmaster'
 
-Note: Hiera uses a systemwide configuration file in ``/etc/puppet/hiera.yaml``
+Hiera uses a systemwide configuration file in ``/etc/puppet/hiera.yaml``
 and this setup supports multiple configurations. The two sets of environments
 that OpenStack Infrastructure uses are ``production`` and ``development``.
-``production`` is the default is and the environment used when nothing else is
+``production`` is the default and the environment used when nothing else is
-specified. Then the configuration needs to be placed into common.yaml in
+specified.
+
+The hiera configuration is placed by puppet apply into common.yaml in
 ``/etc/puppet/hieradata/production`` and ``/etc/puppet/hieradata/development``.
 The values are simple key-value pairs in yaml format. The keys needed are the
 keys referenced in your ``site.pp``, their values are typically obvious
 (strings, lists of strings). ``/etc/puppet/hieradata/`` and below should be
-owned by ``puppet:puppet`` and have mode ``0711``. The actual ``common.yaml``
+owned by ``puppet:puppet`` and have mode ``0711``.
-file should have mode 0600.
+
+Below the ``hieradata`` directory, there should be a ``common.yaml`` file where
+settings that should be available to all servers in the infrastructure go,
+and then two directories full of files. The first is ``fqdn`` which should
+contain a yaml file for every server in the infrastructure named
+``${fqdn_of_server}.yaml``. That file has secrets that are only for that
+server. Additionally, some servers can have a ``$group`` defined in
+``manifests/site.pp``. There can be a correspondingly named yaml file in the
+``group`` directory that contains secrets to be made available to each
+server in the group.
+
+All of the actual yaml files should have mode 0600 and be owned by root.
 
 Adding a node
 -------------
@@ -86,70 +96,48 @@ For manual bootstrap, you need to run on the new server connecting
    wget https://git.openstack.org/cgit/openstack-infra/system-config/plain/install_puppet.sh
    bash -x install_puppet.sh
 
-The node then needs to be configured to set a fixed hostname and the
-hostname of the puppet master with the following additions to
-``/etc/puppet/puppet.conf``:
-
-.. code-block:: ini
-
-   [main]
-   server=puppetmaster.openstack.org
-   certname=review.openstack.org
-
-The cert signing process needs to be started with:
-
-.. code-block:: bash
-
-  sudo puppet agent --test
-
-This will make a request to the puppet master to have its SSL cert signed.
-On the puppet master:
-
-.. code-block:: bash
-
-  sudo puppet cert list
-
-You should get a list of entries similar to the one below::
-
-  review.openstack.org  (44:18:BB:DF:08:50:62:70:17:07:82:1F:D5:70:0E:BF)
-
-If you see the new node there you can sign its cert on the puppet master with:
-
-.. code-block:: bash
-
-  sudo puppet cert sign review.openstack.org
-
-Once the cert is signed, the puppet running orchestration will pick up
-the node and run puppet on it as needed.
-
 Running Puppet on Nodes
 -----------------------
 
 In OpenStack's Infrastructure, puppet runs are triggered from a cronjob
-running on the puppetmaster which in turn runs a single run of puppet on
+running on the puppetmaster which in turn runs a single run of puppet apply on
-each host we know about. We do not use the daemon mode of puppet agent
+each host we know about.
-because it experiences random hangs, and also does not allow us to control
-sequencing in any meaningful way.
 
 The entry point for this process is ``/opt/system-config/production/run_all.sh``
 
-There are a set of nodes, which are configured in puppet as "override" nodes,
+There are a few sets of nodes which have their own playbooks so that they
-which are run in sequence before the rest of the nodes are run in parallel.
+are run in sequence before the rest of the nodes are run in parallel.
 At the moment, this allows creation of git repos on the git slaves before
 creation of the master repos on the gerrit server.
 
+If an admin needs to run puppet by hand, it's a simple matter of either
+logging in to the server in question and running
+`puppet apply /opt/system-config/production/manifests/site.pp` or, on the
+puppetmaster, running
+`ansible-playbook --limit='$HOST;localhost' /opt/system-config/production/playbooks/remote_puppet_all.yaml`
+as root, where `$HOST` is the host you want to run puppet on. If you are
+working with git, gerrit or afs servers, you'll want to replace
+`remote_puppet_all.yaml` with the appropriate specific playbook.
+The `;localhost` is important as some of the plays depend on performing a task
+on the localhost before continuing to the host in question, and without it in
+the limit section, the tasks for the host will have undefined values.
+
+Testing new puppet code can be done via `puppet apply --noop` or by
+constructing a VM with a puppet install in it and just running `puppet apply`
+on the code in question. This should actually make it fairly easy to test
+how production works in a more self-contained manner.
+
+
 Disabling Puppet on Nodes
 -------------------------
 
 In the case of needing to disable the running of puppet on a node, it's a
 simple matter of adding an entry to the ansible inventory "disabled" group.
+See the :ref:`disable-enable-puppet` section for more details.
 
 Important Notes
 ---------------
 
 #. Make sure the site manifest **does not** include the puppet cron job, this
    conflicts with puppet master and can cause issues.  The initial puppet run
-   that create users should be done using the puppet agent configuration above.
+   that create users should be done using the puppet apply configuration above.
-
-#. If you do not see the cert in the master's cert list the agent's
-   ``/var/log/syslog`` should have an entry showing you why.

doc/source/sysadmin.rst

@@ -279,6 +279,8 @@ repository ``https://git.openstack.org/openstack-infra/system-config``. This
 tool is run from a checkout on the puppetmaster - please see :file:`launch/README`
 for detailed instructions.
 
+.. _disable-enable-puppet:
+
 Disable/Enable Puppet
 =====================
 

playbooks/group_vars/all.yaml

@@ -1,4 +1,5 @@
-puppetmaster: puppetmaster.openstack.org
 copy_hieradata: true
 copy_puppet: true
+manifest: /opt/system-config/production/manifests/site.pp
 manifest_base: /opt/system-config
+puppetdb: https://puppetdb.openstack.org:8081