
Script changes to download content by layer. Valid options are 'all', 'compiler', 'distro', 'flock'. Current .lst and yum files under directory centos-mirror-tools are relocated. Generic package dependencies are relocated to centos-mirror-tools/config/<os>/<layer>/ . Lst entries for compilable content have been relocated to other git repos by prior updates. i.e. those that list tarballs or srpms to be compiled within that repo. The original .lst files are deleted to make it easier to identify new content during development. Layer 'all' builds all layers in a single workspace. The lst files are identical to current content, minus the src.rpm and tarball entries. Other layers get only a subset of packages download. The minimum required to build the layer. The 'flock' layer will have additional content to satisfy the run time requirements as well as the build time requirements. An upper layer does not need to list rpms known to be provided by a lower layer. Instead the config file 'required_layer_pkgs.cfg' lists urls for lst files for lower layer build outputs. These build outputs are generated and published by cengn for each layer. A second layer config file, 'required_layer_iso_inc.cfg' lists image.inc files for lower layer builds. These build outputs are generated and published by cengn for each layer, summarizing the image.inc files found in individual git repos. Image.inc files inform the build-iso process, listing rpms that that provide services and commands that need to be included in the iso. The transitive list of required rpms need not be listed. Finally the layer config should include a yum.repos.d directory in which supplementary yum repos are defined to pick up cengn built content from lower layers. To allow a designer to do cross-layer building using local sources rather than those provided by CENGN, there are several options. The designer can modify the urls for lower layer build outputs, as found in the .cfg and .repo files within the config directory 'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>' directly within the git. Substitute urls can use the file:/// syntax. Just be sure to remove these changes before submitting. Alternatively new args have been added to download_mirror.sh, generate-cgcs-centos-repo.sh and commands that override the normal config. The easiest to use is a command argurement that substitutes a new config directory, replacing stx-tools/centos-mirror-tools/config. The intent is for the designer to do a recursive copy of that directory into a side location. make his changes there, outside of git, and provide the path to that directory as an extra arguement to download_mirror.sh and generate-cgcs-centos-repo.sh. e.g. For simplicity I'll only list the 'extra' arguements download_mirror.sh -C <my-config-dir> \ -l <layer> \ ... generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \ --layer=<layer> \ ... populate_downloads.sh --config-dir=<my-config-dir> \ --layer=<layer> \ ... These arguements can also be suplied via the environment. For the purpose of containerized builds, these arguements should be defined in your localrc. e.g. export STX_CONFIG_DIR=<my-config-dir> export LAYER=<layer> The final alternative is to override things at a more granular level, replacing a single lst file of image.inc file. Here you can replace a single line found in a required_layer_pkgs.cfg or required_layer_iso_inc.cfg file. e.g. We are doing a flock build and want to modify the content picked up from the distro layer's rt build, and that content delivers a service we want in the iso. For simplicity I'll only list the 'extra' arguments ./download_mirror.sh -l flock \ -L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \ -I distro,std,file:///<my-distro-workspace>/rt/image.inc \ ... generate-cgcs-centos-repo.sh --layer=flock \ --layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \ --layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \ ... NOTE: The triplet syntax for a package list url is <lower-layer>,<build-type>,<url-to-rpm.lst> lower-layer: 'compiler', 'distro' build-type: 'std', 'rt', 'installer' Also if 'file:///' syntax is used, a matching change is made to the yum *.repo file. This assumes that the rpm.lst is co-resident with repodata directory, as is the norm for our build outputs. NOTE: The triplet syntax for a image inc url is <lower-layer>,<include-type>,<url-to-image.inc> lower-layer: 'compiler', 'distro' build-type: 'std', 'dev' A typical user is likely only working in the flock layer on the master branch. He should be content to use the compiler and distro layer outputs from cengn. His workflow looks like ... 1, sync code for flock layer $ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml $ repo sync 2, download rpms for flock layer, and populate a local mirror $ LOCAL_MIRROR=/import/mirrors/starlingx $ cd stx-tools/centos-mirror-tools $ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ 3, Prepare a virtual repo and downloads directory for building $ cd ../toCOPY $ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR $ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR 4, rpm package and iso building $ build-pkgs && build-iso && build-helm-charts.sh Building all layers in a single workspace is still supported, and looks identical to the previous workflow. 1, sync code $ repo init -u https://opendev.org/starlingx/manifest.git -b master $ repo sync 2, download rpms for flock layer, and populate a local mirror $ LOCAL_MIRROR=/import/mirrors/starlingx $ cd stx-tools/centos-mirror-tools $ ./download_mirror.sh -n -g -c yum.conf.sample -S $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ 3, create repo named "StxCentos7Distro" for building $ cd ../toCOPY $ generate-cgcs-centos-repo.sh $LOCAL_MIRROR $ populate_downloads.sh $LOCAL_MIRROR 4, rpm package and iso building $ build-pkgs && build-iso && build-helm-charts.sh Only a cross-layer developer should setup two or three copies of the building environment, one per layer. We suggest you use seperate shells for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to be unique, Shell 1, compiler layer $ LOCAL_MIRROR=/import/mirrors/starlingx $ LOCAL_CONFIG=<some-dir>/config $ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler $ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root $ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace ... $ mkdir -p $MY_REPO_ROOT_DIR $ cd $MY_REPO_ROOT_DIR $ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml $ cd stx-tools/centos-mirror-tools $ cp -r config/* $LOCAL_CONFIG ... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ... $ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ $ cd ../toCOPY $ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR $ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR $ build-pkgs Shell 2, distro layer $ LOCAL_MIRROR=/import/mirrors/starlingx $ LOCAL_CONFIG=<some-dir>/config $ MY_REPO_ROOT_DIR=<some-dir>/layer-distro $ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root $ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace ... $ mkdir -p $MY_REPO_ROOT_DIR $ cd $MY_REPO_ROOT_DIR $ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml $ repo sync $ cd stx-tools/centos-mirror-tools $ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ $ cd ../toCOPY $ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR $ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR $ build-pkgs Shell 3, flock layer $ LOCAL_MIRROR=/import/mirrors/starlingx $ LOCAL_CONFIG=<some-dir>/config $ MY_REPO_ROOT_DIR=<some-dir>/layer-flock $ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root $ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace ... $ mkdir -p $MY_REPO_ROOT_DIR $ cd $MY_REPO_ROOT_DIR $ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml $ repo sync $ cd stx-tools/centos-mirror-tools $ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ $ cd ../toCOPY $ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR $ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR $ build-pkgs && build-iso && build-helm-charts.sh Story: 2006166 Task: 37103 Depends-On: https://review.opendev.org/698756 Depends-On: https://review.opendev.org/700819 Depends-On: https://review.opendev.org/700821 Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12 Signed-off-by: Scott Little <scott.little@windriver.com>
257 lines
10 KiB
Docker
257 lines
10 KiB
Docker
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
# Copyright (C) 2019 Intel Corporation
|
|
#
|
|
|
|
FROM centos:7.4.1708
|
|
|
|
# Proxy configuration
|
|
#ENV http_proxy "http://your.actual_http_proxy.com:your_port"
|
|
#ENV https_proxy "https://your.actual_https_proxy.com:your_port"
|
|
#ENV ftp_proxy "http://your.actual_ftp_proxy.com:your_port"
|
|
|
|
#RUN echo "proxy=$http_proxy" >> /etc/yum.conf && \
|
|
# echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\
|
|
#export ftp_proxy=$ftp_proxy" >> /root/.bashrc
|
|
|
|
RUN echo "http_caching=packages" >> /etc/yum.conf
|
|
|
|
# username you will docker exec into the container as.
|
|
# It should NOT be your host username so you can easily tell
|
|
# if you are in our out of the container.
|
|
ARG MYUNAME=builder
|
|
ARG MYUID=1000
|
|
|
|
ENV container=docker
|
|
|
|
# Download required dependencies by mirror/build processes.
|
|
# Notice there are 3 invocations to yum package manage.
|
|
# 1) Enable EPEL repository.
|
|
# 2) Download required packages.
|
|
# 3) Clean yum cache.
|
|
RUN groupadd -g 751 cgts && \
|
|
echo "mock:x:751:root" >> /etc/group && \
|
|
echo "mockbuild:x:9001:" >> /etc/group && \
|
|
yum install -y epel-release && \
|
|
yum install -y anaconda \
|
|
anaconda-help \
|
|
anaconda-runtime \
|
|
autoconf-archive \
|
|
autogen \
|
|
automake \
|
|
bc \
|
|
bind \
|
|
bind-utils \
|
|
cpanminus \
|
|
createrepo \
|
|
deltarpm \
|
|
expat-devel \
|
|
isomd5sum \
|
|
gcc \
|
|
gettext \
|
|
git \
|
|
libguestfs-tools \
|
|
libtool \
|
|
libxml2 \
|
|
lighttpd \
|
|
lighttpd-fastcgi \
|
|
lighttpd-mod_geoip \
|
|
net-tools \
|
|
mkisofs \
|
|
mock \
|
|
mongodb \
|
|
mongodb-server \
|
|
pax \
|
|
perl-CPAN \
|
|
python-deltarpm \
|
|
python-pep8 \
|
|
python-pip \
|
|
python-psutil \
|
|
python2-psutil \
|
|
python36-psutil \
|
|
python-sphinx \
|
|
python-subunit \
|
|
python-pip \
|
|
python-testrepository \
|
|
python-tox \
|
|
python-yaml \
|
|
postgresql \
|
|
qemu-kvm \
|
|
quilt \
|
|
rpm-build \
|
|
rpm-sign \
|
|
rpm-python \
|
|
squashfs-tools \
|
|
sudo \
|
|
systemd \
|
|
syslinux \
|
|
syslinux-utils \
|
|
udisks2 \
|
|
vim-enhanced \
|
|
wget \
|
|
yumdownloader
|
|
|
|
# This image requires a set of scripts and helpers
|
|
# for working correctly, in this section they are
|
|
# copied inside the image.
|
|
COPY toCOPY/finishSetup.sh /usr/local/bin
|
|
COPY toCOPY/populate_downloads.sh /usr/local/bin
|
|
COPY toCOPY/generate-cgcs-tis-repo /usr/local/bin
|
|
COPY toCOPY/generate-cgcs-centos-repo.sh /usr/local/bin
|
|
COPY toCOPY/lst_utils.sh /usr/local/bin
|
|
COPY toCOPY/.inputrc /home/$MYUNAME/
|
|
COPY toCOPY/builder-constraints.txt /home/$MYUNAME/
|
|
|
|
# cpan modules, installing with cpanminus to avoid stupid questions since cpan is whack
|
|
RUN cpanm --notest Fatal && \
|
|
cpanm --notest XML::SAX && \
|
|
cpanm --notest XML::SAX::Expat && \
|
|
cpanm --notest XML::Parser && \
|
|
cpanm --notest XML::Simple
|
|
|
|
# pip installs
|
|
RUN pip install -c /home/$MYUNAME/builder-constraints.txt python-subunit junitxml --upgrade && \
|
|
pip install -c /home/$MYUNAME/builder-constraints.txt tox --upgrade
|
|
|
|
# Install repo tool
|
|
RUN curl https://storage.googleapis.com/git-repo-downloads/repo > /usr/local/bin/repo && \
|
|
chmod a+x /usr/local/bin/repo
|
|
|
|
# installing go and setting paths
|
|
ENV GOPATH="/usr/local/go"
|
|
ENV PATH="${GOPATH}/bin:${PATH}"
|
|
RUN yum install -y golang && \
|
|
mkdir -p ${GOPATH}/bin && \
|
|
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
|
|
|
|
# mock time
|
|
# forcing chroots since a couple of packages naughtily insist on network access and
|
|
# we dont have nspawn and networks happy together.
|
|
RUN useradd -s /sbin/nologin -u 9001 -g 9001 mockbuild && \
|
|
rmdir /var/lib/mock && \
|
|
ln -s /localdisk/loadbuild/mock /var/lib/mock && \
|
|
rmdir /var/cache/mock && \
|
|
ln -s /localdisk/loadbuild/mock-cache /var/cache/mock && \
|
|
echo "config_opts['use_nspawn'] = False" >> /etc/mock/site-defaults.cfg && \
|
|
echo "config_opts['rpmbuild_networking'] = True" >> /etc/mock/site-defaults.cfg && \
|
|
echo >> /etc/mock/site-defaults.cfg
|
|
|
|
# Inherited tools for mock stuff
|
|
# we at least need the mock_cache_unlock tool
|
|
# they install into /usr/bin
|
|
COPY toCOPY/mock_overlay /opt/mock_overlay
|
|
RUN cd /opt/mock_overlay && \
|
|
make && \
|
|
make install
|
|
|
|
# ENV setup
|
|
RUN echo "# Load stx-builder configuration" >> /etc/profile.d/TC.sh && \
|
|
echo "if [[ -r \${HOME}/buildrc ]]; then" >> /etc/profile.d/TC.sh && \
|
|
echo " source \${HOME}/buildrc" >> /etc/profile.d/TC.sh && \
|
|
echo " export PROJECT SRC_BUILD_ENVIRONMENT MYPROJECTNAME MYUNAME" >> /etc/profile.d/TC.sh && \
|
|
echo " export MY_BUILD_CFG MY_BUILD_CFG_RT MY_BUILD_CFG_STD MY_BUILD_DIR MY_BUILD_ENVIRONMENT MY_BUILD_ENVIRONMENT_FILE MY_BUILD_ENVIRONMENT_FILE_RT MY_BUILD_ENVIRONMENT_FILE_STD MY_DEBUG_BUILD_CFG_RT MY_DEBUG_BUILD_CFG_STD MY_LOCAL_DISK MY_MOCK_ROOT MY_REPO MY_REPO_ROOT_DIR MY_SRC_RPM_BUILD_DIR MY_TC_RELEASE MY_WORKSPACE LAYER" >> /etc/profile.d/TC.sh && \
|
|
echo "fi" >> /etc/profile.d/TC.sh && \
|
|
echo "export FORMAL_BUILD=0" >> /etc/profile.d/TC.sh && \
|
|
echo "export PATH=\$MY_REPO/build-tools:\$PATH" >> /etc/profile.d/TC.sh
|
|
|
|
# centos locales are broken. this needs to be run after the last yum install/update
|
|
RUN localedef -i en_US -f UTF-8 en_US.UTF-8
|
|
|
|
# setup
|
|
RUN mkdir -p /www/run && \
|
|
mkdir -p /www/logs && \
|
|
mkdir -p /www/home && \
|
|
mkdir -p /www/root/htdocs/localdisk && \
|
|
chown -R $MYUID:cgts /www && \
|
|
ln -s /localdisk/loadbuild /www/root/htdocs/localdisk/loadbuild && \
|
|
ln -s /import/mirrors/CentOS /www/root/htdocs/CentOS && \
|
|
ln -s /import/mirrors/fedora /www/root/htdocs/fedora && \
|
|
ln -s /localdisk/designer /www/root/htdocs/localdisk/designer
|
|
|
|
# lighthttpd setup
|
|
# chmod for /var/log/lighttpd fixes a centos issue
|
|
# in place sed for server root since it's expanded soon thereafter
|
|
# echo "server.bind = \"localhost\"" >> /etc/lighttpd/lighttpd.conf && \
|
|
RUN echo "$MYUNAME ALL=(ALL:ALL) NOPASSWD:ALL" >> /etc/sudoers && \
|
|
mkdir -p /var/log/lighttpd && \
|
|
chmod a+rwx /var/log/lighttpd/ && \
|
|
sed -i 's%^var\.log_root.*$%var.log_root = "/www/logs"%g' /etc/lighttpd/lighttpd.conf && \
|
|
sed -i 's%^var\.server_root.*$%var.server_root = "/www/root"%g' /etc/lighttpd/lighttpd.conf && \
|
|
sed -i 's%^var\.home_dir.*$%var.home_dir = "/www/home"%g' /etc/lighttpd/lighttpd.conf && \
|
|
sed -i 's%^var\.state_dir.*$%var.state_dir = "/www/run"%g' /etc/lighttpd/lighttpd.conf && \
|
|
sed -i "s/server.port/#server.port/g" /etc/lighttpd/lighttpd.conf && \
|
|
sed -i "s/server.use-ipv6/#server.use-ipv6/g" /etc/lighttpd/lighttpd.conf && \
|
|
sed -i "s/server.username/#server.username/g" /etc/lighttpd/lighttpd.conf && \
|
|
sed -i "s/server.groupname/#server.groupname/g" /etc/lighttpd/lighttpd.conf && \
|
|
sed -i "s/server.bind/#server.bind/g" /etc/lighttpd/lighttpd.conf && \
|
|
sed -i "s/server.document-root/#server.document-root/g" /etc/lighttpd/lighttpd.conf && \
|
|
sed -i "s/server.dirlisting/#server.dirlisting/g" /etc/lighttpd/lighttpd.conf && \
|
|
echo "server.port = 8088" >> /etc/lighttpd/lighttpd.conf && \
|
|
echo "server.use-ipv6 = \"disable\"" >> /etc/lighttpd/lighttpd.conf && \
|
|
echo "server.username = \"$MYUNAME\"" >> /etc/lighttpd/lighttpd.conf && \
|
|
echo "server.groupname = \"cgts\"" >> /etc/lighttpd/lighttpd.conf && \
|
|
echo "server.bind = \"localhost\"" >> /etc/lighttpd/lighttpd.conf && \
|
|
echo "server.document-root = \"/www/root/htdocs\"" >> /etc/lighttpd/lighttpd.conf && \
|
|
sed -i "s/dir-listing.activate/#dir-listing.activate/g" /etc/lighttpd/conf.d/dirlisting.conf && \
|
|
echo "dir-listing.activate = \"enable\"" >> /etc/lighttpd/conf.d/dirlisting.conf
|
|
|
|
# Systemd Enablement
|
|
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
|
|
rm -f /lib/systemd/system/multi-user.target.wants/*;\
|
|
rm -f /etc/systemd/system/*.wants/*;\
|
|
rm -f /lib/systemd/system/local-fs.target.wants/*; \
|
|
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
|
|
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
|
|
rm -f /lib/systemd/system/basic.target.wants/*;\
|
|
rm -f /lib/systemd/system/anaconda.target.wants/*
|
|
|
|
# Without this, init won't start the enabled services and exec'ing and starting
|
|
# them reports "Failed to get D-Bus connection: Operation not permitted".
|
|
VOLUME /run /tmp
|
|
|
|
RUN useradd -r -u $MYUID -g cgts -m $MYUNAME && \
|
|
ln -s /home/$MYUNAME/.ssh /mySSH && \
|
|
rsync -av /etc/skel/ /home/$MYUNAME/
|
|
|
|
# now that we are doing systemd, make the startup script be in bashrc
|
|
# also we need to SHADOW the udev centric mkefiboot script with a sudo centric one
|
|
RUN echo "bash -C /usr/local/bin/finishSetup.sh" >> /home/$MYUNAME/.bashrc && \
|
|
echo "export PATH=/usr/local/bin:/localdisk/designer/$MYUNAME/bin:\$PATH" >> /home/$MYUNAME/.bashrc && \
|
|
chmod a+x /usr/local/bin/*
|
|
|
|
# Genrate a git configuration file in order to save an extra step
|
|
# for end users, this file is required by "repo" tool.
|
|
RUN chown $MYUNAME /home/$MYUNAME && \
|
|
runuser -u $MYUNAME -- git config --global user.email $MYUNAME@starlingx.com && \
|
|
runuser -u $MYUNAME -- git config --global user.name $MYUNAME && \
|
|
runuser -u $MYUNAME -- git config --global color.ui false
|
|
|
|
# Customizations for mirror creation
|
|
RUN rm /etc/yum.repos.d/CentOS-Sources.repo
|
|
RUN rm /etc/yum.repos.d/epel.repo
|
|
COPY centos-mirror-tools/yum.repos.d/* /etc/yum.repos.d/
|
|
COPY centos-mirror-tools/rpm-gpg-keys/* /etc/pki/rpm-gpg/
|
|
|
|
# Import GPG keys
|
|
RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY*
|
|
|
|
# Try to continue a yum command even if a StarlingX repo is unavailable.
|
|
RUN yum-config-manager --setopt=StarlingX\*.skip_if_unavailable=1 --save
|
|
|
|
# When we run 'init' below, it will run systemd, and systemd requires RTMIN+3
|
|
# to exit cleanly. By default, docker stop uses SIGTERM, which systemd ignores.
|
|
STOPSIGNAL RTMIN+3
|
|
|
|
# Don't know if it's possible to run services without starting this
|
|
CMD /usr/sbin/init
|