diff --git a/doc/source/logstash.rst b/doc/source/logstash.rst deleted file mode 100644 index 630ec57fbd..0000000000 --- a/doc/source/logstash.rst +++ /dev/null @@ -1,293 +0,0 @@ -:title: Logstash - -.. _logstash: - -Logstash -######## - -Logstash is a high-performance indexing and search engine for logs. - -At a Glance -=========== - -:Hosts: - * http://logstash.openstack.org - * logstash-worker\*.openstack.org - * elasticsearch\*.openstack.org -:Puppet: - * https://opendev.org/opendev/puppet-logstash - * :git_file:`modules/openstack_project/manifests/logstash.pp` - * :git_file:`modules/openstack_project/manifests/logstash_worker.pp` - * :git_file:`modules/openstack_project/manifests/elasticsearch.pp` -:Configuration: - * :git_file:`modules/openstack_project/files/logstash` - * :git_file:`modules/openstack_project/templates/logstash` - * `submit-logstash-jobs defaults`_ -:Projects: - * http://logstash.net/ - * http://kibana.org/ - * http://www.elasticsearch.org/ - * http://crm114.sourceforge.net/ -:Bugs: - * https://storyboard.openstack.org/#!/project/748 - * https://logstash.jira.com/secure/Dashboard.jspa - * https://github.com/rashidkpc/Kibana/issues - * https://github.com/elasticsearch/elasticsearch/issues - -Overview -======== - -Logs from Zuul test runs are sent to logstash where they are -indexed and stored. Logstash facilitates reviewing logs from multiple -sources in a single test run, searching for errors or particular -events within a test run, as well as searching for log event trends -across test runs. - -System Architecture -=================== - -There are four major layers in our Logstash setup. - -1. Submit Logstash Jobs. - The `logs post-playbook`_ in the Zuul ``base`` job submit logs defined - in the `submit-logstash-jobs defaults`_ to a Logstash Indexer. -2. Logstash Indexer. - Reads these log events from the log pusher, filters them to remove - unwanted lines, collapses multiline events together, and parses - useful information out of the events before shipping them to - ElasticSearch for storage and indexing. -3. ElasticSearch. - Provides log storage, indexing, and search. -4. Kibana. - A Logstash oriented web client for ElasticSearch. You can perform - queries on your Logstash logs in ElasticSearch through Kibana using - the Lucene query language. - -Each layer scales horizontally. As the number of logs grows we can add -more log pushers, more Logstash indexers, and more ElasticSearch nodes. -Currently we have multiple Logstash worker nodes that pair a log pusher -with a Logstash indexer. We did this as each Logstash process can only -dedicate a single thread to filtering log events which turns into a -bottleneck very quickly. This looks something like: - -:: - - zuul post-logs playbook - | - | - gearman-client - / | \ - / | \ - gearman gearman gearman - worker1 worker2 worker3 - | | | - logstash logstash logstash - indexer1 indexer2 indexer3 - \ | / - \ | / - elasticsearch - cluster - | - | - kibana - -Log Pusher ----------- - -This is an ansible module in the `submit-log-processor-jobs role`_. It -submits Gearman jobs to push log files into logstash. - -Log pushing looks like this: - -* Zuul runs post-playbook on job completion. -* Using info in the Gearman job log files are retrieved. -* Log files are processed then shipped to Logstash. - -Using Gearman allows us to scale the number of log pushers -horizontally. It is as simple as adding another process that talks to -the Gearman server. - -If you are interested in technical details the source of these scripts -can be found at - -* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-client.py -* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py - -Logstash --------- - -Logstash does the heavy lifting of squashing all of our log lines into -events with a common format. It reads the JSON log events from the log -pusher connected to it, deletes events we don't want, parses log lines -to set the timestamp, message, and other fields for the event, then -ships these processed events off to ElasticSearch where they are stored -and made queryable. - -At a high level Logstash takes: - -:: - - { - "fields" { - "build_name": "gate-foo", - "build_numer": "10", - "event_message": "2013-05-31T17:31:39.113 DEBUG Something happened", - }, - } - -And turns that into: - -:: - - { - "fields" { - "build_name": "gate-foo", - "build_numer": "10", - "loglevel": "DEBUG" - }, - "@message": "Something happened", - "@timestamp": "2013-05-31T17:31:39.113Z", - } - -It flattens each log line into something that looks very much like -all of the other events regardless of the source log line format. This -makes querying your logs for lines from a specific build that failed -between two timestamps with specific message content very easy. You -don't need to write complicated greps instead you query against a -schema. - -The config file that tells Logstash how to do this flattening can be -found at -https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf - -This works via the tags that are associated with a given message. - -The tags in -https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf -are used to tell logstash how to parse a given file's messages, based -on the file's message format. - -When adding a new file to be indexed to -https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml -at least one tag from the openstack-filters.conf file should be associated -with the new file. One can expect to see '{%logmessage%}' instead of -actual message data if indexing is not working properly. - -In the event a new file's format is not covered, a patch for -https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf -should be submitted with an appropriate parsing pattern. - -ElasticSearch -------------- - -ElasticSearch is basically a REST API layer for Lucene. It provides -the storage and search engine for Logstash. It scales horizontally and -loves it when you give it more memory. Currently we run a multi-node -cluster on large VMs to give ElasticSearch both memory and disk space. -Per index (Logstash creates one index per day) we have N+1 replica -redundancy to distribute disk utilization and provide high availability. -Each replica is broken into multiple shards providing increased indexing -and search throughput as each shard is essentially a valid mini index. - -To check on the cluster health, run this command on any es.* node:: - - curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' - -Kibana ------- - -Kibana is a ruby app sitting behind Apache that provides a nice web UI -for querying Logstash events stored in ElasticSearch. Our install can -be reached at http://logstash.openstack.org. See -:ref:`query-logstash` for more info on using Kibana to perform -queries. - -simpleproxy ------------ -Simpleproxy is a simple tcp proxy which allows forwarding tcp connections from -one host to another. We use it to forward mysql traffic from a publicly -accessible host to the trove instance running the subunit2sql MySQL DB. This -allows for public access to the data on the database through the host -logstash.openstack.org. - -.. _query-logstash: - -Querying Logstash -================= - -Hop on over to http://logstash.openstack.org and by default you get the -last 15 minutes of everything Logstash knows about in chunks of 100. -We run a lot of tests but it is possible no logs have come in over the -last 15 minutes, change the dropdown in the top left from ``Last 15m`` -to ``Last 60m`` to get a better window on the logs. At this point you -should see a list of logs, if you click on a log event it will expand -and show you all of the fields associated with that event and their -values (note Chromium and Kibana seem to have trouble with this at times -and some fields end up without values, use Firefox if this happens). -You can search based on all of these fields and if you click the -magnifying glass next to a field in the expanded event view it will add -that field and value to your search. This is a good way of refining -searches without a lot of typing. - -The above is good info for poking around in the Logstash logs, but -one of your changes has a failing test and you want to know why. We -can jumpstart the refining process with a simple query. - -``@fields.build_change:"$FAILING_CHANGE" AND @fields.build_patchset:"$FAILING_PATCHSET" AND @fields.build_name:"$FAILING_BUILD_NAME" AND @fields.build_number:"$FAILING_BUILD_NUMBER"`` - -This will show you all logs available from the patchset and build pair -that failed. Chances are that this is still a significant number of -logs and you will want to do more filtering. You can add more filters -to the query using ``AND`` and ``OR`` and parentheses can be used to -group sections of the query. Potential additions to the above query -might be - -* ``AND @fields.filename:"logs/syslog.txt"`` to get syslog events. -* ``AND @fields.filename:"logs/screen-n-api.txt"`` to get Nova API events. -* ``AND @fields.loglevel:"ERROR"`` to get ERROR level events. -* ``AND @message"error"`` to get events with error in their message. - and so on. - -General query tips: - -* Don't search ``All time``. ElasticSearch is bad at trying to find all - the things it ever knew about. Give it a window of time to look - through. You can use the presets in the dropdown to select a window or - use the ``foo`` to ``bar`` boxes above the frequency graph. -* Only the @message field can have fuzzy searches performed on it. Other - fields require specific information. -* This system is growing fast and may not always keep up with the load. - Be patient. If expected logs do not show up immediately after the - Zuul job completes wait a few minutes. - -crm114 -======= - -In an effort to assist with automated failure detection, the infra team -has started leveraging crm114 to classify and analyze the messages stored -by logstash. - -The tool utilizes a statistical approach for classifying data, and is -frequently used as an email spam detector. For logstash data, the idea -is to flag those log entries that are not in passing runs and only in -failing ones, which should be useful in pinpointing what caused the -failures. - -In the OpenStack logstash system, crm114 attaches an error_pr attribute -to all indexed entries. Values from -1000.00 to -10.00 should be considered -sufficient to get all potential errors as identified by the program. -Used in a kibana query, it would be structured like this: - -:: - - error_pr:["-1000.0" TO "-10.0"] - - -This is still an early effort and additional tuning and refinement should -be expected. Should the crm114 settings need to be tuned or expanded, -a patch may be submitted for this file, which controls the process: -https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/classify-log.crm - -.. _logs post-playbook: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml -.. _submit-logstash-jobs defaults: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml -.. _submit-log-processor-jobs role: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-log-processor-jobs diff --git a/doc/source/systems.rst b/doc/source/systems.rst index b6c7c75171..666df70b80 100644 --- a/doc/source/systems.rst +++ b/doc/source/systems.rst @@ -16,7 +16,6 @@ Major Systems grafyaml keycloak zuul - logstash elastic-recheck devstack-gate nodepool diff --git a/hiera/common.yaml b/hiera/common.yaml index 5eadefe33a..bdfadefe87 100644 --- a/hiera/common.yaml +++ b/hiera/common.yaml @@ -14,12 +14,6 @@ cacti_hosts: - bridge.openstack.org - cacti.openstack.org - eavesdrop01.opendev.org -- elasticsearch02.openstack.org -- elasticsearch03.openstack.org -- elasticsearch04.openstack.org -- elasticsearch05.openstack.org -- elasticsearch06.openstack.org -- elasticsearch07.openstack.org - ethercalc02.openstack.org - etherpad01.opendev.org - gitea-lb01.opendev.org @@ -39,27 +33,6 @@ cacti_hosts: - kdc04.openstack.org - keycloak01.opendev.org - lists.openstack.org -- logstash-worker01.openstack.org -- logstash-worker02.openstack.org -- logstash-worker03.openstack.org -- logstash-worker04.openstack.org -- logstash-worker05.openstack.org -- logstash-worker06.openstack.org -- logstash-worker07.openstack.org -- logstash-worker08.openstack.org -- logstash-worker09.openstack.org -- logstash-worker10.openstack.org -- logstash-worker11.openstack.org -- logstash-worker12.openstack.org -- logstash-worker13.openstack.org -- logstash-worker14.openstack.org -- logstash-worker15.openstack.org -- logstash-worker16.openstack.org -- logstash-worker17.openstack.org -- logstash-worker18.openstack.org -- logstash-worker19.openstack.org -- logstash-worker20.openstack.org -- logstash.openstack.org - nb01.opendev.org - nb02.opendev.org - nb03.opendev.org diff --git a/inventory/base/hosts.yaml b/inventory/base/hosts.yaml index e0785a9b3f..de8396f375 100644 --- a/inventory/base/hosts.yaml +++ b/inventory/base/hosts.yaml @@ -91,48 +91,6 @@ all: region_name: DFW public_v4: 104.239.144.232 public_v6: 2001:4800:7818:104:be76:4eff:fe04:46c8 - elasticsearch02.openstack.org: - ansible_host: 104.130.159.19 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 104.130.159.19 - public_v6: 2001:4800:7818:102:be76:4eff:fe05:40b - elasticsearch03.openstack.org: - ansible_host: 104.130.246.141 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 104.130.246.141 - public_v6: 2001:4800:7819:103:be76:4eff:fe05:f26 - elasticsearch04.openstack.org: - ansible_host: 104.130.246.138 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 104.130.246.138 - public_v6: 2001:4800:7819:103:be76:4eff:fe02:3042 - elasticsearch05.openstack.org: - ansible_host: 104.130.159.15 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 104.130.159.15 - public_v6: 2001:4800:7818:102:be76:4eff:fe04:55b5 - elasticsearch06.openstack.org: - ansible_host: 104.130.246.111 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 104.130.246.111 - public_v6: 2001:4800:7819:103:be76:4eff:fe04:b9d7 - elasticsearch07.openstack.org: - ansible_host: 104.130.246.44 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 104.130.246.44 - public_v6: 2001:4800:7819:103:be76:4eff:fe04:35bb ethercalc02.openstack.org: ansible_host: 162.242.144.125 location: @@ -279,153 +237,6 @@ all: region_name: DFW public_v4: 50.56.173.222 public_v6: 2001:4800:780e:510:3bc3:d7f6:ff04:b736 - logstash-worker01.openstack.org: - ansible_host: 172.99.116.137 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 172.99.116.137 - public_v6: 2001:4800:7821:105:be76:4eff:fe04:b9e7 - logstash-worker02.openstack.org: - ansible_host: 23.253.253.66 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.253.66 - public_v6: 2001:4800:7817:104:be76:4eff:fe04:aee1 - logstash-worker03.openstack.org: - ansible_host: 23.253.242.246 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.242.246 - public_v6: 2001:4800:7817:104:be76:4eff:fe04:81e6 - logstash-worker04.openstack.org: - ansible_host: 166.78.47.27 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 166.78.47.27 - public_v6: 2001:4800:7817:101:be76:4eff:fe04:46ec - logstash-worker05.openstack.org: - ansible_host: 23.253.100.127 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.100.127 - public_v6: 2001:4800:7817:101:be76:4eff:fe04:3b27 - logstash-worker06.openstack.org: - ansible_host: 23.253.100.168 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.100.168 - public_v6: 2001:4800:7817:101:be76:4eff:fe04:5e21 - logstash-worker07.openstack.org: - ansible_host: 172.99.116.187 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 172.99.116.187 - public_v6: 2001:4800:7821:105:be76:4eff:fe04:bca2 - logstash-worker08.openstack.org: - ansible_host: 166.78.47.61 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 166.78.47.61 - public_v6: 2001:4800:7817:101:be76:4eff:fe04:432c - logstash-worker09.openstack.org: - ansible_host: 23.253.119.179 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.119.179 - public_v6: 2001:4800:7817:103:be76:4eff:fe04:38d1 - logstash-worker10.openstack.org: - ansible_host: 172.99.116.111 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 172.99.116.111 - public_v6: 2001:4800:7821:105:be76:4eff:fe04:3a04 - logstash-worker11.openstack.org: - ansible_host: 166.78.47.86 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 166.78.47.86 - public_v6: 2001:4800:7817:101:be76:4eff:fe05:3e1 - logstash-worker12.openstack.org: - ansible_host: 166.78.174.249 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 166.78.174.249 - public_v6: 2001:4800:7815:105:be76:4eff:fe01:61e4 - logstash-worker13.openstack.org: - ansible_host: 23.253.230.28 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.230.28 - public_v6: 2001:4800:7817:103:be76:4eff:fe04:6ff1 - logstash-worker14.openstack.org: - ansible_host: 172.99.116.170 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 172.99.116.170 - public_v6: 2001:4800:7821:105:be76:4eff:fe04:4a4a - logstash-worker15.openstack.org: - ansible_host: 23.253.242.14 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.242.14 - public_v6: 2001:4800:7817:104:be76:4eff:fe04:829b - logstash-worker16.openstack.org: - ansible_host: 23.253.230.58 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.230.58 - public_v6: 2001:4800:7817:103:be76:4eff:fe03:5f45 - logstash-worker17.openstack.org: - ansible_host: 172.99.117.120 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 172.99.117.120 - public_v6: 2001:4800:7821:105:be76:4eff:fe04:6173 - logstash-worker18.openstack.org: - ansible_host: 172.99.116.203 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 172.99.116.203 - public_v6: 2001:4800:7821:105:be76:4eff:fe05:435 - logstash-worker19.openstack.org: - ansible_host: 166.78.47.173 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 166.78.47.173 - public_v6: 2001:4800:7817:101:be76:4eff:fe03:a5ec - logstash-worker20.openstack.org: - ansible_host: 23.253.76.170 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 23.253.76.170 - public_v6: 2001:4800:7815:105:be76:4eff:fe04:59a6 - logstash01.openstack.org: - ansible_host: 104.239.141.90 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 104.239.141.90 - public_v6: 2001:4800:7819:104:be76:4eff:fe04:935b meetpad01.opendev.org: ansible_host: 104.239.240.194 location: diff --git a/inventory/service/group_vars/elasticsearch.yaml b/inventory/service/group_vars/elasticsearch.yaml deleted file mode 100644 index 942bfb1e7b..0000000000 --- a/inventory/service/group_vars/elasticsearch.yaml +++ /dev/null @@ -1,4 +0,0 @@ -iptables_extra_allowed_groups: - - {'protocol': 'tcp', 'port': '9200:9400', 'group': 'elasticsearch'} - - {'protocol': 'tcp', 'port': '9200:9400', 'group': 'logstash'} - - {'protocol': 'tcp', 'port': '9200:9400', 'group': 'logstash-worker'} diff --git a/inventory/service/group_vars/graphite.yaml b/inventory/service/group_vars/graphite.yaml index 028e179a5f..25ff99962a 100644 --- a/inventory/service/group_vars/graphite.yaml +++ b/inventory/service/group_vars/graphite.yaml @@ -8,7 +8,6 @@ iptables_extra_allowed_hosts: iptables_extra_allowed_groups: - {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'} - - {'protocol': 'udp', 'port': '8125', 'group': 'logstash'} - {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'} - {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'} - {'protocol': 'udp', 'port': '8125', 'group': 'zuul'} diff --git a/inventory/service/group_vars/graphite_opendev.org b/inventory/service/group_vars/graphite_opendev.org index 25753b7eed..07b4e83f71 100644 --- a/inventory/service/group_vars/graphite_opendev.org +++ b/inventory/service/group_vars/graphite_opendev.org @@ -8,7 +8,6 @@ iptables_extra_allowed_hosts: iptables_extra_allowed_groups: - {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'} - - {'protocol': 'udp', 'port': '8125', 'group': 'logstash'} - {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'} - {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'} - {'protocol': 'udp', 'port': '8125', 'group': 'zuul'} diff --git a/inventory/service/group_vars/logstash.yaml b/inventory/service/group_vars/logstash.yaml deleted file mode 100644 index 4295a978c1..0000000000 --- a/inventory/service/group_vars/logstash.yaml +++ /dev/null @@ -1,6 +0,0 @@ -iptables_extra_public_tcp_ports: - - 80 - - 3306 -iptables_extra_allowed_groups: - - {'protocol': 'tcp', 'port': '4730', 'group': 'logstash-worker'} - - {'protocol': 'tcp', 'port': '4730', 'group': 'zuul-executor'} diff --git a/inventory/service/groups.yaml b/inventory/service/groups.yaml index e2037f961f..61a3714e29 100644 --- a/inventory/service/groups.yaml +++ b/inventory/service/groups.yaml @@ -52,7 +52,6 @@ groups: - adns*.opendev.org - ns*.opendev.org eavesdrop: eavesdrop[0-9]*.opendev.org - elasticsearch: elasticsearch[0-9]*.open*.org ethercalc: ethercalc*.open*.org etherpad: etherpad[0-9]*.open*.org gitea: @@ -105,10 +104,6 @@ groups: - storyboard[0-9]*.opendev.org - translate[0-9]*.open*.org - zuul[0-9]*.opendev.org - logstash: - - logstash[0-9]*.open*.org - logstash-worker: - - logstash-worker[0-9]*.open*.org mailman: - lists*.katacontainers.io - lists*.open*.org @@ -131,11 +126,8 @@ groups: - paste[0-9]*.opendev.org puppet: - cacti[0-9]*.open*.org - - elasticsearch[0-9]*.open*.org - ethercalc[0-9]*.open*.org - health[0-9]*.openstack.org - - logstash-worker[0-9]*.open*.org - - logstash[0-9]*.open*.org - status*.open*.org - storyboard-dev[0-9]*.opendev.org - storyboard[0-9]*.opendev.org @@ -143,11 +135,8 @@ groups: - translate[0-9]*.open*.org puppet4: - cacti[0-9]*.open*.org - - elasticsearch[0-9]*.open*.org - ethercalc[0-9]*.open*.org - health[0-9]*.openstack.org - - logstash-worker[0-9]*.open*.org - - logstash[0-9]*.open*.org - status*.open*.org - storyboard[0-9]*.opendev.org - storyboard-dev[0-9]*.opendev.org diff --git a/manifests/site.pp b/manifests/site.pp index 41fe4d31d0..865d887175 100644 --- a/manifests/site.pp +++ b/manifests/site.pp @@ -20,66 +20,6 @@ node /^ethercalc\d+\.open.*\.org$/ { } } -# Node-OS: xenial -node /^logstash\d*\.open.*\.org$/ { - class { 'openstack_project::server': } - - class { 'openstack_project::logstash': - discover_nodes => [ - 'elasticsearch03.openstack.org:9200', - 'elasticsearch04.openstack.org:9200', - 'elasticsearch05.openstack.org:9200', - 'elasticsearch06.openstack.org:9200', - 'elasticsearch07.openstack.org:9200', - 'elasticsearch02.openstack.org:9200', - ], - subunit2sql_db_host => hiera('subunit2sql_db_host', ''), - subunit2sql_db_pass => hiera('subunit2sql_db_password', ''), - } -} - -# Node-OS: xenial -node /^logstash-worker\d+\.open.*\.org$/ { - $group = 'logstash-worker' - - $elasticsearch_nodes = [ - 'elasticsearch02.openstack.org', - 'elasticsearch03.openstack.org', - 'elasticsearch04.openstack.org', - 'elasticsearch05.openstack.org', - 'elasticsearch06.openstack.org', - 'elasticsearch07.openstack.org', - ] - - class { 'openstack_project::server': } - - class { 'openstack_project::logstash_worker': - discover_node => 'elasticsearch03.openstack.org', - enable_mqtt => false, - mqtt_password => hiera('mqtt_service_user_password'), - mqtt_ca_cert_contents => hiera('mosquitto_tls_ca_file'), - } -} - -# Node-OS: xenial -node /^elasticsearch\d+\.open.*\.org$/ { - $group = "elasticsearch" - - $elasticsearch_nodes = [ - 'elasticsearch02.openstack.org', - 'elasticsearch03.openstack.org', - 'elasticsearch04.openstack.org', - 'elasticsearch05.openstack.org', - 'elasticsearch06.openstack.org', - 'elasticsearch07.openstack.org', - ] - - class { 'openstack_project::server': } - class { 'openstack_project::elasticsearch_node': - discover_nodes => $elasticsearch_nodes, - } -} - # A machine to run Storyboard # Node-OS: xenial node /^storyboard\d+\.opendev\.org$/ { diff --git a/modules.env b/modules.env index 6d37e45ef8..1a08fadcff 100644 --- a/modules.env +++ b/modules.env @@ -58,15 +58,12 @@ SOURCE_MODULES["https://github.com/voxpupuli/puppet-nodejs"]="v2.3.0" # Please keep sorted INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-bup"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elastic_recheck"]="origin/master" -INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elasticsearch"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ethercalc"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-httpd"]="origin/master" # Storyboard and translate use the jeepyb module INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-jeepyb"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-kibana"]="origin/master" -INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-log_processor"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logrotate"]="origin/master" -INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logstash"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-mysql_backup"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-pip"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-project_config"]="origin/master" @@ -76,7 +73,6 @@ INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-reviewday"]="origin/mast INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-simpleproxy"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ssh"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-storyboard"]="origin/master" -INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-subunit2sql"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-tmpreaper"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ulimit"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-user"]="origin/master" diff --git a/modules/openstack_project/files/logstash/logstash-indexer.default b/modules/openstack_project/files/logstash/logstash-indexer.default deleted file mode 100644 index 57f6819aac..0000000000 --- a/modules/openstack_project/files/logstash/logstash-indexer.default +++ /dev/null @@ -1,3 +0,0 @@ -# Increase the max heap size to twice the default. -# Default is 25% of memory or 1g whichever is less. -JAVA_ARGS='-Xmx2g' diff --git a/modules/openstack_project/manifests/elasticsearch_node.pp b/modules/openstack_project/manifests/elasticsearch_node.pp deleted file mode 100644 index adacb39c60..0000000000 --- a/modules/openstack_project/manifests/elasticsearch_node.pp +++ /dev/null @@ -1,46 +0,0 @@ -# Copyright 2013 Hewlett-Packard Development Company, L.P. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. -# -# Elasticsearch server glue class. -# -class openstack_project::elasticsearch_node ( - $discover_nodes = ['localhost'], - $heap_size = '30g', -) { - class { 'logstash::elasticsearch': } - - class { '::elasticsearch': - es_template_config => { - 'index.store.compress.stored' => true, - 'index.store.compress.tv' => true, - 'indices.memory.index_buffer_size' => '33%', - 'indices.breaker.fielddata.limit' => '70%', - 'bootstrap.mlockall' => true, - 'gateway.recover_after_nodes' => '5', - 'gateway.recover_after_time' => '5m', - 'gateway.expected_nodes' => '6', - 'discovery.zen.minimum_master_nodes' => '4', - 'discovery.zen.ping.multicast.enabled' => false, - 'discovery.zen.ping.unicast.hosts' => $discover_nodes, - 'http.cors.enabled' => true, - 'http.cors.allow-origin' => "'*'", # lint:ignore:double_quoted_strings - }, - heap_size => $heap_size, - version => '1.7.6', - } - - class { 'logstash::curator': - keep_for_days => '7', - } -} diff --git a/modules/openstack_project/manifests/logstash.pp b/modules/openstack_project/manifests/logstash.pp deleted file mode 100644 index 49e410b2e2..0000000000 --- a/modules/openstack_project/manifests/logstash.pp +++ /dev/null @@ -1,47 +0,0 @@ -# Copyright 2013 Hewlett-Packard Development Company, L.P. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. -# -# Logstash web frontend glue class. -# -class openstack_project::logstash ( - $discover_nodes = ['elasticsearch02.openstack.org:9200'], - $statsd_host = 'graphite.opendev.org', - $subunit2sql_db_host, - $subunit2sql_db_pass, -) { - class { 'logstash::web': - frontend => 'kibana', - discover_nodes => $discover_nodes, - proxy_elasticsearch => true, - } - - class { 'log_processor': } - - class { 'log_processor::geard': - statsd_host => $statsd_host, - } - - include 'subunit2sql' - - class { 'subunit2sql::server': - db_host => $subunit2sql_db_host, - db_pass => $subunit2sql_db_pass, - } - - include 'simpleproxy' - - class { 'simpleproxy::server': - db_host => $subunit2sql_db_host, - } -} diff --git a/modules/openstack_project/manifests/logstash_worker.pp b/modules/openstack_project/manifests/logstash_worker.pp deleted file mode 100644 index 9a8dda914e..0000000000 --- a/modules/openstack_project/manifests/logstash_worker.pp +++ /dev/null @@ -1,100 +0,0 @@ -# Copyright 2013 Hewlett-Packard Development Company, L.P. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. -# -# Logstash indexer worker glue class. -# -class openstack_project::logstash_worker ( - $discover_node = 'elasticsearch02.openstack.org', - $filter_rev = 'master', - $filter_source = 'https://opendev.org/openstack/logstash-filters', - $enable_mqtt = true, - $mqtt_hostname = 'firehose.openstack.org', - $mqtt_port = 8883, - $mqtt_topic = "logstash/${::hostname}", - $mqtt_username = 'infra', - $mqtt_password = undef, - $mqtt_ca_cert_contents = undef, -) { - - file { '/etc/logprocessor/worker.yaml': - ensure => present, - owner => 'root', - group => 'root', - mode => '0644', - content => template('openstack_project/logstash/jenkins-log-worker.yaml.erb'), - require => Class['::log_processor'], - } - - file { '/etc/default/logstash-indexer': - ensure => present, - owner => 'root', - group => 'root', - mode => '0644', - source => 'puppet:///modules/openstack_project/logstash/logstash-indexer.default', - } - - vcsrepo { '/opt/logstash-filters': - ensure => latest, - provider => git, - revision => $filter_rev, - source => $filter_source, - } - - include ::logstash - - logstash::filter { 'openstack-logstash-filters': - level => '50', - target => '/opt/logstash-filters/filters/openstack-filters.conf', - require => [ - Class['::logstash'], - Vcsrepo['/opt/logstash-filters'], - ], - notify => Service['logstash'], - } - - file { '/etc/logstash/mqtt-root-CA.pem.crt': - ensure => present, - content => $mqtt_ca_cert_contents, - replace => true, - owner => 'root', - group => 'root', - mode => '0555', - require => Class['::logstash'], - } - - validate_array($elasticsearch_nodes) # needed by output.conf.erb - class { '::logstash::indexer': - input_template => 'openstack_project/logstash/input.conf.erb', - output_template => 'openstack_project/logstash/output.conf.erb', - require => Logstash::Filter['openstack-logstash-filters'], - } - - include ::log_processor - log_processor::worker { 'A': - config_file => '/etc/logprocessor/worker.yaml', - require => File['/etc/logprocessor/worker.yaml'], - } - log_processor::worker { 'B': - config_file => '/etc/logprocessor/worker.yaml', - require => File['/etc/logprocessor/worker.yaml'], - } - log_processor::worker { 'C': - config_file => '/etc/logprocessor/worker.yaml', - require => File['/etc/logprocessor/worker.yaml'], - } - log_processor::worker { 'D': - config_file => '/etc/logprocessor/worker.yaml', - require => File['/etc/logprocessor/worker.yaml'], - } -} diff --git a/modules/openstack_project/templates/logstash/input.conf.erb b/modules/openstack_project/templates/logstash/input.conf.erb deleted file mode 100644 index 7e748a26d7..0000000000 --- a/modules/openstack_project/templates/logstash/input.conf.erb +++ /dev/null @@ -1,8 +0,0 @@ -input { - tcp { - host => "localhost" - port => 9999 - codec => json_lines {} - type => "jenkins" - } -} diff --git a/modules/openstack_project/templates/logstash/jenkins-log-worker.yaml.erb b/modules/openstack_project/templates/logstash/jenkins-log-worker.yaml.erb deleted file mode 100644 index 263b5e24c3..0000000000 --- a/modules/openstack_project/templates/logstash/jenkins-log-worker.yaml.erb +++ /dev/null @@ -1,13 +0,0 @@ -gearman-host: logstash.openstack.org -gearman-port: 4730 -output-host: localhost -output-port: 9999 -output-mode: tcp -crm114-script: /usr/local/bin/classify-log.crm -crm114-data: /var/lib/crm114 -mqtt-host: <%= @mqtt_hostname %> -mqtt-port: <%= @mqtt_port %> -mqtt-topic: gearman-logstash/<%= @hostname %> -mqtt-user: <%= @mqtt_username %> -mqtt-pass: <%= @mqtt_password %> -mqtt-ca-certs: /etc/logstash/mqtt-root-CA.pem.crt diff --git a/modules/openstack_project/templates/logstash/output.conf.erb b/modules/openstack_project/templates/logstash/output.conf.erb deleted file mode 100644 index fd6db1fdea..0000000000 --- a/modules/openstack_project/templates/logstash/output.conf.erb +++ /dev/null @@ -1,9 +0,0 @@ -output { - elasticsearch { - hosts => <%= @elasticsearch_nodes.map { |node| node + ":9200" }.inspect %> - manage_template => false - flush_size => 1024 - timeout => 300 - } - -} diff --git a/modules/openstack_project/templates/logstash/redis-output.conf.erb b/modules/openstack_project/templates/logstash/redis-output.conf.erb deleted file mode 100644 index f02e500001..0000000000 --- a/modules/openstack_project/templates/logstash/redis-output.conf.erb +++ /dev/null @@ -1,3 +0,0 @@ -output { - redis { host => "127.0.0.1" data_type => "list" key => "logstash" } -} diff --git a/playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/results.yaml b/playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/results.yaml index 2b2bdc9560..e4b8eea43c 100644 --- a/playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/results.yaml +++ b/playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/results.yaml @@ -22,11 +22,6 @@ results: - letsencrypt - mailman - logstash-worker02.openstack.org: - - logstash-worker - - puppet - - puppet4 - mirror02.regionone.linaro-us.opendev.org: - afs-client - kerberos-client diff --git a/zuul.d/puppet.yaml b/zuul.d/puppet.yaml index b26182b368..fda5759dc1 100644 --- a/zuul.d/puppet.yaml +++ b/zuul.d/puppet.yaml @@ -11,14 +11,11 @@ - opendev/ansible-role-puppet - opendev/puppet-bup - opendev/puppet-elastic_recheck - - opendev/puppet-elasticsearch - opendev/puppet-ethercalc - opendev/puppet-httpd - opendev/puppet-jeepyb - opendev/puppet-kibana - - opendev/puppet-log_processor - opendev/puppet-logrotate - - opendev/puppet-logstash - opendev/puppet-mysql_backup - opendev/puppet-openstack_infra_spec_helper - opendev/puppet-pip @@ -28,7 +25,6 @@ - opendev/puppet-simpleproxy - opendev/puppet-ssh - opendev/puppet-storyboard - - opendev/puppet-subunit2sql - opendev/puppet-tmpreaper - opendev/puppet-ulimit - opendev/puppet-user @@ -87,19 +83,15 @@ - opendev/puppet-project_config - opendev/puppet-ethercalc - opendev/puppet-httpd - - opendev/puppet-subunit2sql - opendev/puppet-reviewday - opendev/puppet-kibana - opendev/puppet-redis - opendev/puppet-zanata - - opendev/puppet-logstash - opendev/puppet-tmpreaper - opendev/puppet-elastic_recheck - opendev/puppet-ulimit - opendev/puppet-logrotate - - opendev/puppet-elasticsearch - opendev/puppet-storyboard - - opendev/puppet-log_processor - opendev/puppet-simpleproxy - opendev/puppet-bup - opendev/puppet-ssh