Cloud-init based factory installation

This commit introduces a systematic approach for performing
standalone installations tailored for factory environments by:

1. Implementing a factory-install framework -
services with staged execution to facilitate the
operations necessary for installing, bootstrapping, and
deploying a site with factory defaults.

2. Leveraging cloud-init to stage necessary configurations
(using default cloud-init nocloud seed data) and to initiate
the factory installation process.

Overall, key features include:
- Configuration: Sets up factory default user and network
- Factory Staging: Sets up necessary services, configurations
  and scripts for factory installation
- Service Execution: Utilizes systemd service for
  executing various installation stages:
    - Bootstrap: Executes the Ansible bootstrap playbook
    - Config: Executes Ansible deployment manager playbook
    - Setup: System setup scripts are triggered post-
      deployment to ensure deployment is reconciled
    - Tests: Test scripts are executed post-deployment
      to validate system with health checks
- Localized Installation: Ensures factory installation
  processes are streamlined for autonomous running, focusing on
  local execution (no external dependencies) without site-specific
  configurations

Test Plan:
1. PASS: Verify successful end-to-end factory installation.
         Ensure the System in the unlocked-enabled-available state.
         Furthermore, ensure:
          1) No alarms raised
          2) System/host reconciled
          3) No errors reported in /var/log/factory-install
2. PASS: Verify cloud-int factory default user and network setup
         - Sysadmin user password and OAM Network configured
         - Ensure no errors reported in /var/log/cloud-init.log
           and /var/log/cloud-init-output.log
3. PASS: Verify factory install staging, ensure:
         1) /var/lib/factory-install created with
           setup, tests, scripts and config contents
         2) Config contents copied to HOME dir
         3) factory-install systemd path, service, target units
           copied to /etc/systemd/system/
         4) /var/lib/factory-install/enabled flag set at start
         5) factory-install.target service triggered
         6) /var/lib/factory-install/state/final flag set upon
            completion
4. PASS: Verify factory install services execution, ensure:
         1) Streamlined staged execution. At most 1
            factory install service unit must be active at a time,
            strictly in the following order:
            bootstrap, config, setup and test
         2) All 4 path units must be active at start, with each
            stage transition, the previous path+service
            unit must go inactive
         3) Intended service states pre/post host unlock
         4) Intended service state after forced system reboot
         5) /var/lib/factory-install/{stage,state}/ has all 4
            flags set upon completion, and no flags at the start
5. PASS: Validate standalone bootstrap and deployment
         (no network service dependencies)
6. PASS: Verify hardware check step
7. PASS: Verify test stage executed

Story: 2011100
Task: 49972

Depends-on: https://review.opendev.org/c/starlingx/metal/+/914995
Depends-on: https://review.opendev.org/c/starlingx/utilities/+/914993

Co-authored-by: Kyle MacLeod <kyle.macleod@windriver.com>
Change-Id: I63a5e62bcd119ccbefc35184afee4c102821995c
Signed-off-by: Salman Rana <salman.rana@windriver.com>

tools/nocloud-factory-install/factory-install/config/localhost.yml

@@ -0,0 +1,8 @@
+# Ansible Bootstrap Configurations - user configuration overrides
+#
+# System and network bootstrap properties cen be specified here as outlined in
+# the StarlingX documentation. For instance, OAM network configuration can
+# be specified: external_oam_subnet, external_oam_gateway_address,
+# external_oam_floating_address, etc.
+
+system_mode: simplex

tools/nocloud-factory-install/factory-install/scripts/10-init-setup

@@ -0,0 +1,69 @@
+#!/bin/bash
+#
+# Copyright (c) 2024 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# cloud-init script to setup the factory install services
+#
+
+USER=sysadmin
+HOME=/home/${USER}
+NOCLOUD=/opt/nocloud
+
+FACTORY_INSTALL=/var/lib/factory-install
+
+check_rc_die() {
+    local -i rc=${1}
+    msg=${2}
+    if [ ${rc} -ne 0 ]; then
+        echo "FATAL: ${msg} [rc=${rc}]" && exit 1
+    fi
+}
+
+echo "Factory Install Setup - Start"
+
+if [ -d "${FACTORY_INSTALL}" ]; then
+    echo "${FACTORY_INSTALL} exists, aborting"
+    exit 1
+fi
+
+mkdir -p "${FACTORY_INSTALL}"/{stage,state}
+check_rc_die $? "mkdir failed"
+
+# Copy system config, setup and test scripts
+cp -r "${NOCLOUD}"/factory-install/scripts "${FACTORY_INSTALL}"/scripts && \
+    cp -r "${NOCLOUD}"/factory-install/setup "${FACTORY_INSTALL}"/setup && \
+    cp -r "${NOCLOUD}"/factory-install/tests "${FACTORY_INSTALL}"/tests && \
+    cp -r "${NOCLOUD}"/factory-install/config "${FACTORY_INSTALL}"/config
+check_rc_die $? "copy failed"
+
+# Ensure files are executable for run-parts
+chmod a+x "${FACTORY_INSTALL}"/scripts/* && \
+    chmod a+x "${FACTORY_INSTALL}"/setup/* && \
+    chmod a+x "${FACTORY_INSTALL}"/tests/*
+check_rc_die $? "chmod failed"
+
+# Copy configuration files required for running bootstrap and deployment configuration services
+# NOTE: Configuration files are expected to be located in home directory
+su "${USER}" <<EOF
+if [ "$(ls "${FACTORY_INSTALL}"/config)" ]; then
+    cp -r "${FACTORY_INSTALL}"/config/* "${HOME}"
+fi
+EOF
+
+echo "Factory Install Setup - Initialize systemd services"
+
+mkdir -p /etc/systemd/system/factory-install.target.wants
+check_rc_die $? "mkdir failed (factory-install.target.wants)"
+
+cp "${NOCLOUD}"/factory-install/systemd/*.{path,service,target} /etc/systemd/system/
+check_rc_die $? "Copy failed (systemd path,service,target)"
+
+cp "${NOCLOUD}"/factory-install/systemd/20-factory-install.preset /etc/systemd/system-preset/ && \
+    chmod a+x /etc/systemd/system-preset/20-factory-install.preset
+check_rc_die $? "Copy failed (systemd preset)"
+
+echo "Factory Install Setup - Complete"
+
+exit 0

tools/nocloud-factory-install/factory-install/scripts/20-hardware-check

@@ -0,0 +1,24 @@
+#!/bin/bash
+#
+# Copyright (c) 2024 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# cloud-init script to Perform hardware and firmware checks
+# TODO: Sample only. Replace with real hardware checks
+
+echo "Hardware Check - Start"
+
+BOARD_VENDOR=$(cat /sys/devices/virtual/dmi/id/board_vendor)
+BOARD_NAME=$(cat /sys/devices/virtual/dmi/id/board_name)
+PRODUCT_NAME=$(cat /sys/devices/virtual/dmi/id/product_name)
+BIOS_VERSION=$(cat /sys/devices/virtual/dmi/id/bios_version)
+
+echo "BOARD_VENDOR=${BOARD_VENDOR}"
+echo "BOARD_NAME=${BOARD_NAME}"
+echo "PRODUCT_NAME=${PRODUCT_NAME}"
+echo "BIOS_VERISON=${BIOS_VERISON}"
+
+echo "Hardware Check - Complete"
+
+exit 0

tools/nocloud-factory-install/factory-install/scripts/90-init-final

@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+# Copyright (c) 2024 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# cloud-init script to Perform hardware and firmware checks
+# TODO: Sample only. Replace with real hardware checks
+#
+
+FACTORY_INSTALL=/var/lib/factory-install
+
+echo "Factory Install Final - Start"
+
+touch "${FACTORY_INSTALL}"/enabled
+
+systemctl daemon-reload
+systemctl preset-all
+
+# Trigger first stage (bootstrap) of factory install services
+touch ${FACTORY_INSTALL}/stage/bootstrap
+
+echo "Factory Install Final - Complete"
+
+systemctl start factory-install.target
+
+exit 0

tools/nocloud-factory-install/factory-install/setup/10-system-setup

@@ -0,0 +1,29 @@
+#!/bin/bash
+#
+# Copyright (c) 2024 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# Factory install system setup triggered during the config stage
+#
+
+echo "System Setup - Start"
+
+echo "Wait - host goenabled"
+until [ -f /var/run/goenabled ]; do
+    sleep 10
+done
+echo "Ready - host goenabled"
+
+echo "Wait - system deployment reconciled"
+while true; do
+    SYSTEM_RECONCILED=$(kubectl --kubeconfig=/etc/kubernetes/admin.conf -n deployment get system -o jsonpath='{.items[0].status.reconciled}')
+    HOST_RECONCILED=$(kubectl --kubeconfig=/etc/kubernetes/admin.conf -n deployment get host controller-0 -o jsonpath='{.status.reconciled}')
+    if [ $SYSTEM_RECONCILED = true -a $HOST_RECONCILED = true ]; then break; fi
+    sleep 10
+done
+echo "Ready - system deployment reconciled"
+
+echo "System Setup - Complete"
+
+exit 0

tools/nocloud-factory-install/factory-install/systemd/20-factory-install.preset

@@ -0,0 +1,10 @@
+enable factory-install.target
+
+enable factory-install-bootstrap.path
+enable factory-install-bootstrap.service
+enable factory-install-config.path
+enable factory-install-config.service
+enable factory-install-setup.path
+enable factory-install-setup.service
+enable factory-install-tests.path
+enable factory-install-tests.service

tools/nocloud-factory-install/factory-install/systemd/factory-install-bootstrap.path

@@ -0,0 +1,10 @@
+[Unit]
+Description=Factory Installation Bootstrap Stage
+ConditionPathExists=/var/lib/factory-install/enabled
+ConditionPathExists=!/var/lib/factory-install/state/bootstrap
+
+[Path]
+PathExists=/var/lib/factory-install/stage/bootstrap
+
+[Install]
+WantedBy=factory-install.target

tools/nocloud-factory-install/factory-install/systemd/factory-install-bootstrap.service

@@ -0,0 +1,20 @@
+[Unit]
+Description=Factory Installation Bootstrap
+ConditionPathExists=/var/lib/factory-install/enabled
+ConditionPathExists=/var/lib/factory-install/stage/bootstrap
+ConditionPathExists=!/var/lib/factory-install/state/bootstrap
+
+[Service]
+Type=oneshot
+User=sysadmin
+ExecStart=/usr/bin/ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml
+ExecStartPost=+/usr/bin/touch /var/lib/factory-install/state/bootstrap
+ExecStartPost=+/usr/bin/touch /var/lib/factory-install/stage/config
+StandardOutput=append:/var/log/factory-install.log
+RemainAfterExit=yes
+Restart=no
+TimeoutStartSec=infinity
+TasksMax=infinity
+
+[Install]
+WantedBy=factory-install.target

tools/nocloud-factory-install/factory-install/systemd/factory-install-config.path

@@ -0,0 +1,10 @@
+[Unit]
+Description=Factory Installation Config Stage
+ConditionPathExists=/var/lib/factory-install/enabled
+ConditionPathExists=!/var/lib/factory-install/state/config
+
+[Path]
+PathExists=/var/lib/factory-install/stage/config
+
+[Install]
+WantedBy=factory-install.target

tools/nocloud-factory-install/factory-install/systemd/factory-install-config.service

@@ -0,0 +1,21 @@
+[Unit]
+Description=Factory Installation Deployment Configuration
+ConditionPathExists=/var/lib/factory-install/enabled
+ConditionPathExists=/var/lib/factory-install/stage/config
+ConditionPathExists=!/var/lib/factory-install/state/config
+After=factory-install-bootstrap.service
+
+[Service]
+Type=oneshot
+User=sysadmin
+ExecStart=/usr/bin/ansible-playbook /usr/local/share/applications/playbooks/wind-river-cloud-platform-deployment-manager.yaml -e "@/home/sysadmin/dm-playbook-overrides.yaml"
+ExecStartPost=+/usr/bin/touch /var/lib/factory-install/state/config
+ExecStartPost=+/usr/bin/touch /var/lib/factory-install/stage/setup
+StandardOutput=append:/var/log/factory-install.log
+RemainAfterExit=yes
+Restart=no
+TimeoutStartSec=infinity
+TasksMax=infinity
+
+[Install]
+WantedBy=factory-install.target

tools/nocloud-factory-install/factory-install/systemd/factory-install-setup.path

@@ -0,0 +1,10 @@
+[Unit]
+Description=Factory Installation System Setup Stage
+ConditionPathExists=/var/lib/factory-install/enabled
+ConditionPathExists=!/var/lib/factory-install/state/setup
+
+[Path]
+PathExists=/var/lib/factory-install/stage/setup
+
+[Install]
+WantedBy=factory-install.target

tools/nocloud-factory-install/factory-install/systemd/factory-install-setup.service

@@ -0,0 +1,21 @@
+[Unit]
+Description=Factory Installation Execute System Setup
+ConditionPathExists=/var/lib/factory-install/enabled
+ConditionPathExists=/var/lib/factory-install/stage/setup
+ConditionPathExists=!/var/lib/factory-install/state/setup
+After=factory-install-config.service
+
+[Service]
+Type=oneshot
+User=sysadmin
+ExecStart=/usr/bin/run-parts --verbose --exit-on-error /var/lib/factory-install/setup
+ExecStartPost=+/usr/bin/touch /var/lib/factory-install/state/setup
+ExecStartPost=+/usr/bin/touch /var/lib/factory-install/stage/tests
+StandardOutput=append:/var/log/factory-install.log
+RemainAfterExit=yes
+Restart=no
+TimeoutStartSec=infinity
+TasksMax=infinity
+
+[Install]
+WantedBy=factory-install.target

tools/nocloud-factory-install/factory-install/systemd/factory-install-tests.path

@@ -0,0 +1,10 @@
+[Unit]
+Description=Factory Installation System Tests Stage
+ConditionPathExists=/var/lib/factory-install/enabled
+ConditionPathExists=!/var/lib/factory-install/state/tests
+
+[Path]
+PathExists=/var/lib/factory-install/stage/tests
+
+[Install]
+WantedBy=factory-install.target

tools/nocloud-factory-install/factory-install/systemd/factory-install-tests.service

@@ -0,0 +1,22 @@
+[Unit]
+Description=Factory Installation Execute System Tests
+ConditionPathExists=/var/lib/factory-install/enabled
+ConditionPathExists=/var/lib/factory-install/stage/tests
+ConditionPathExists=!/var/lib/factory-install/state/tests
+After=factory-install-setup.service
+
+[Service]
+Type=oneshot
+User=sysadmin
+ExecStart=/usr/bin/run-parts --verbose --exit-on-error /var/lib/factory-install/tests
+ExecStartPost=+/usr/bin/touch /var/lib/factory-install/state/tests
+ExecStartPost=+/usr/bin/touch /var/lib/factory-install/stage/final
+ExecStartPost=+/usr/bin/rm /var/lib/factory-install/enabled
+StandardOutput=append:/var/log/factory-install.log
+RemainAfterExit=yes
+Restart=no
+TimeoutStartSec=infinity
+TasksMax=infinity
+
+[Install]
+WantedBy=factory-install.target

tools/nocloud-factory-install/factory-install/systemd/factory-install.target

@@ -0,0 +1,7 @@
+# Target enabled by cloud-init
+[Unit]
+Description=Factory Install Target
+After=cloud-init.target network-online.target
+
+[Install]
+WantedBy=multi-user.target

tools/nocloud-factory-install/factory-install/tests/10-system-health

@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+# Copyright (c) 2024 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# Factory install system health checks triggered during the tests stage
+#
+
+
+echo "System Health Checks - Start"
+
+fail () {
+    echo "FAIL: $1" && exit 1
+}
+
+# check for service impacting alarms
+# TODO update alarm check - currently checks for any alarms
+source /etc/platform/openrc
+fm --timeout 10 alarm-list --nowrap|grep -e "major\|minor\|warning\|critical"
+if [ $? == 0 ]; then
+    fail "service impacting alarms present"
+fi
+
+echo "System Health Checks - Complete"
+
+exit 0

tools/nocloud-factory-install/meta-data

@@ -0,0 +1,2 @@
+instance-id: iid-local01
+local-hostname: controller-0

tools/nocloud-factory-install/network-config

@@ -0,0 +1,21 @@
+# Cloud-init networking config.
+#
+# "This network configuration format lets users customise their instance’s networking
+# interfaces by assigning subnet configuration, virtual device creation (bonds, bridges, VLANs)
+# routes and DNS configuration." [1]
+#
+# This config is meant to be a placeholder. It should be
+# updated based on the factory node networking requirements.
+#
+#  Offical cloutinit documentation with explination of the various file params can be found at:
+# [1] https://cloudinit.readthedocs.io/en/latest/reference/network-config-format-v1.html
+#
+version: 1
+config:
+   - type: physical
+     name: enp2s1
+     subnets:
+        - type: static
+          address: 10.10.10.2
+          netmask: 255.255.255.0
+          gateway: 10.10.10.1

tools/nocloud-factory-install/user-data

@@ -0,0 +1,14 @@
+#cloud-config
+
+chpasswd:
+  list:
+    # Changes the sysadmin password - the hash below specifies St8rlingX*1234
+    - sysadmin:$5$HElSMXRZZ8wlTiEe$I0hValcFqxLRKm3pFdXrpGZlxnmzQt6i9lhIR9FWAf8
+  expire: False
+
+runcmd:
+  - [ /bin/bash, -c, "echo $(date): Initiating factory-install" ]
+  - mkdir -p /opt/nocloud
+  - mount LABEL=CIDATA /opt/nocloud
+  - run-parts --verbose --exit-on-error /opt/nocloud/factory-install/scripts
+  - eject /opt/nocloud