tools/Dockerfile
Scott Little d51f8050cd Build layering, script and lst update
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.

Current .lst and yum files under directory centos-mirror-tools
are relocated.  Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .

Lst entries for compilable content have been relocated to other
git repos by prior updates.  i.e.  those that list tarballs or
srpms to be compiled within that repo.

The original .lst files are deleted to make it easier
to identify new content during development.

Layer 'all' builds all layers in a single workspace.  The
lst files are identical to current content, minus the src.rpm and
tarball entries.

Other layers get only a subset of packages download. The minimum
required to build the layer.  The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.

An upper layer does not need to list rpms known to be provided by
a lower layer.  Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.

A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds.  These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.

Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso.  The transitive list of required rpms need not be listed.

Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.

To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.

The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git.  Substitute urls can use the file:///
syntax.  Just be sure to remove these changes before submitting.

Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.

The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.

e.g. For simplicity I'll only list the 'extra' arguements

download_mirror.sh -C <my-config-dir> \
                   -l <layer> \
                   ...

generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
                             --layer=<layer> \
                             ...

populate_downloads.sh --config-dir=<my-config-dir> \
                      --layer=<layer> \
                      ...

These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.

e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>

The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file.  Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.

e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso.  For simplicity I'll only list the 'extra' arguments

./download_mirror.sh -l flock \
    -L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    -I distro,std,file:///<my-distro-workspace>/rt/image.inc \
    ...

generate-cgcs-centos-repo.sh --layer=flock \
    --layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    --layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
    ...

NOTE: The triplet syntax for a package list url is
    <lower-layer>,<build-type>,<url-to-rpm.lst>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'rt', 'installer'

    Also if 'file:///' syntax is used, a matching change is made to
    the yum *.repo file.  This assumes that the rpm.lst is co-resident with
    repodata directory, as is the norm for our build outputs.

NOTE: The triplet syntax for a image inc url is
    <lower-layer>,<include-type>,<url-to-image.inc>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'dev'

A typical user is likely only working in the flock layer on the master
branch.  He should be content to use the compiler and distro layer
outputs from cengn.

His workflow looks like ...

1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.

1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Only a cross-layer developer should setup two or three copies of the
building environment, one per layer.  We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,

Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...

$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
 ... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...

$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh

Story: 2006166
Task: 37103

Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-02-10 10:45:40 -05:00

257 lines
10 KiB
Docker

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Copyright (C) 2019 Intel Corporation
#
FROM centos:7.4.1708
# Proxy configuration
#ENV http_proxy "http://your.actual_http_proxy.com:your_port"
#ENV https_proxy "https://your.actual_https_proxy.com:your_port"
#ENV ftp_proxy "http://your.actual_ftp_proxy.com:your_port"
#RUN echo "proxy=$http_proxy" >> /etc/yum.conf && \
# echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\
#export ftp_proxy=$ftp_proxy" >> /root/.bashrc
RUN echo "http_caching=packages" >> /etc/yum.conf
# username you will docker exec into the container as.
# It should NOT be your host username so you can easily tell
# if you are in our out of the container.
ARG MYUNAME=builder
ARG MYUID=1000
ENV container=docker
# Download required dependencies by mirror/build processes.
# Notice there are 3 invocations to yum package manage.
# 1) Enable EPEL repository.
# 2) Download required packages.
# 3) Clean yum cache.
RUN groupadd -g 751 cgts && \
echo "mock:x:751:root" >> /etc/group && \
echo "mockbuild:x:9001:" >> /etc/group && \
yum install -y epel-release && \
yum install -y anaconda \
anaconda-help \
anaconda-runtime \
autoconf-archive \
autogen \
automake \
bc \
bind \
bind-utils \
cpanminus \
createrepo \
deltarpm \
expat-devel \
isomd5sum \
gcc \
gettext \
git \
libguestfs-tools \
libtool \
libxml2 \
lighttpd \
lighttpd-fastcgi \
lighttpd-mod_geoip \
net-tools \
mkisofs \
mock \
mongodb \
mongodb-server \
pax \
perl-CPAN \
python-deltarpm \
python-pep8 \
python-pip \
python-psutil \
python2-psutil \
python36-psutil \
python-sphinx \
python-subunit \
python-pip \
python-testrepository \
python-tox \
python-yaml \
postgresql \
qemu-kvm \
quilt \
rpm-build \
rpm-sign \
rpm-python \
squashfs-tools \
sudo \
systemd \
syslinux \
syslinux-utils \
udisks2 \
vim-enhanced \
wget \
yumdownloader
# This image requires a set of scripts and helpers
# for working correctly, in this section they are
# copied inside the image.
COPY toCOPY/finishSetup.sh /usr/local/bin
COPY toCOPY/populate_downloads.sh /usr/local/bin
COPY toCOPY/generate-cgcs-tis-repo /usr/local/bin
COPY toCOPY/generate-cgcs-centos-repo.sh /usr/local/bin
COPY toCOPY/lst_utils.sh /usr/local/bin
COPY toCOPY/.inputrc /home/$MYUNAME/
COPY toCOPY/builder-constraints.txt /home/$MYUNAME/
# cpan modules, installing with cpanminus to avoid stupid questions since cpan is whack
RUN cpanm --notest Fatal && \
cpanm --notest XML::SAX && \
cpanm --notest XML::SAX::Expat && \
cpanm --notest XML::Parser && \
cpanm --notest XML::Simple
# pip installs
RUN pip install -c /home/$MYUNAME/builder-constraints.txt python-subunit junitxml --upgrade && \
pip install -c /home/$MYUNAME/builder-constraints.txt tox --upgrade
# Install repo tool
RUN curl https://storage.googleapis.com/git-repo-downloads/repo > /usr/local/bin/repo && \
chmod a+x /usr/local/bin/repo
# installing go and setting paths
ENV GOPATH="/usr/local/go"
ENV PATH="${GOPATH}/bin:${PATH}"
RUN yum install -y golang && \
mkdir -p ${GOPATH}/bin && \
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
# mock time
# forcing chroots since a couple of packages naughtily insist on network access and
# we dont have nspawn and networks happy together.
RUN useradd -s /sbin/nologin -u 9001 -g 9001 mockbuild && \
rmdir /var/lib/mock && \
ln -s /localdisk/loadbuild/mock /var/lib/mock && \
rmdir /var/cache/mock && \
ln -s /localdisk/loadbuild/mock-cache /var/cache/mock && \
echo "config_opts['use_nspawn'] = False" >> /etc/mock/site-defaults.cfg && \
echo "config_opts['rpmbuild_networking'] = True" >> /etc/mock/site-defaults.cfg && \
echo >> /etc/mock/site-defaults.cfg
# Inherited tools for mock stuff
# we at least need the mock_cache_unlock tool
# they install into /usr/bin
COPY toCOPY/mock_overlay /opt/mock_overlay
RUN cd /opt/mock_overlay && \
make && \
make install
# ENV setup
RUN echo "# Load stx-builder configuration" >> /etc/profile.d/TC.sh && \
echo "if [[ -r \${HOME}/buildrc ]]; then" >> /etc/profile.d/TC.sh && \
echo " source \${HOME}/buildrc" >> /etc/profile.d/TC.sh && \
echo " export PROJECT SRC_BUILD_ENVIRONMENT MYPROJECTNAME MYUNAME" >> /etc/profile.d/TC.sh && \
echo " export MY_BUILD_CFG MY_BUILD_CFG_RT MY_BUILD_CFG_STD MY_BUILD_DIR MY_BUILD_ENVIRONMENT MY_BUILD_ENVIRONMENT_FILE MY_BUILD_ENVIRONMENT_FILE_RT MY_BUILD_ENVIRONMENT_FILE_STD MY_DEBUG_BUILD_CFG_RT MY_DEBUG_BUILD_CFG_STD MY_LOCAL_DISK MY_MOCK_ROOT MY_REPO MY_REPO_ROOT_DIR MY_SRC_RPM_BUILD_DIR MY_TC_RELEASE MY_WORKSPACE LAYER" >> /etc/profile.d/TC.sh && \
echo "fi" >> /etc/profile.d/TC.sh && \
echo "export FORMAL_BUILD=0" >> /etc/profile.d/TC.sh && \
echo "export PATH=\$MY_REPO/build-tools:\$PATH" >> /etc/profile.d/TC.sh
# centos locales are broken. this needs to be run after the last yum install/update
RUN localedef -i en_US -f UTF-8 en_US.UTF-8
# setup
RUN mkdir -p /www/run && \
mkdir -p /www/logs && \
mkdir -p /www/home && \
mkdir -p /www/root/htdocs/localdisk && \
chown -R $MYUID:cgts /www && \
ln -s /localdisk/loadbuild /www/root/htdocs/localdisk/loadbuild && \
ln -s /import/mirrors/CentOS /www/root/htdocs/CentOS && \
ln -s /import/mirrors/fedora /www/root/htdocs/fedora && \
ln -s /localdisk/designer /www/root/htdocs/localdisk/designer
# lighthttpd setup
# chmod for /var/log/lighttpd fixes a centos issue
# in place sed for server root since it's expanded soon thereafter
# echo "server.bind = \"localhost\"" >> /etc/lighttpd/lighttpd.conf && \
RUN echo "$MYUNAME ALL=(ALL:ALL) NOPASSWD:ALL" >> /etc/sudoers && \
mkdir -p /var/log/lighttpd && \
chmod a+rwx /var/log/lighttpd/ && \
sed -i 's%^var\.log_root.*$%var.log_root = "/www/logs"%g' /etc/lighttpd/lighttpd.conf && \
sed -i 's%^var\.server_root.*$%var.server_root = "/www/root"%g' /etc/lighttpd/lighttpd.conf && \
sed -i 's%^var\.home_dir.*$%var.home_dir = "/www/home"%g' /etc/lighttpd/lighttpd.conf && \
sed -i 's%^var\.state_dir.*$%var.state_dir = "/www/run"%g' /etc/lighttpd/lighttpd.conf && \
sed -i "s/server.port/#server.port/g" /etc/lighttpd/lighttpd.conf && \
sed -i "s/server.use-ipv6/#server.use-ipv6/g" /etc/lighttpd/lighttpd.conf && \
sed -i "s/server.username/#server.username/g" /etc/lighttpd/lighttpd.conf && \
sed -i "s/server.groupname/#server.groupname/g" /etc/lighttpd/lighttpd.conf && \
sed -i "s/server.bind/#server.bind/g" /etc/lighttpd/lighttpd.conf && \
sed -i "s/server.document-root/#server.document-root/g" /etc/lighttpd/lighttpd.conf && \
sed -i "s/server.dirlisting/#server.dirlisting/g" /etc/lighttpd/lighttpd.conf && \
echo "server.port = 8088" >> /etc/lighttpd/lighttpd.conf && \
echo "server.use-ipv6 = \"disable\"" >> /etc/lighttpd/lighttpd.conf && \
echo "server.username = \"$MYUNAME\"" >> /etc/lighttpd/lighttpd.conf && \
echo "server.groupname = \"cgts\"" >> /etc/lighttpd/lighttpd.conf && \
echo "server.bind = \"localhost\"" >> /etc/lighttpd/lighttpd.conf && \
echo "server.document-root = \"/www/root/htdocs\"" >> /etc/lighttpd/lighttpd.conf && \
sed -i "s/dir-listing.activate/#dir-listing.activate/g" /etc/lighttpd/conf.d/dirlisting.conf && \
echo "dir-listing.activate = \"enable\"" >> /etc/lighttpd/conf.d/dirlisting.conf
# Systemd Enablement
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*
# Without this, init won't start the enabled services and exec'ing and starting
# them reports "Failed to get D-Bus connection: Operation not permitted".
VOLUME /run /tmp
RUN useradd -r -u $MYUID -g cgts -m $MYUNAME && \
ln -s /home/$MYUNAME/.ssh /mySSH && \
rsync -av /etc/skel/ /home/$MYUNAME/
# now that we are doing systemd, make the startup script be in bashrc
# also we need to SHADOW the udev centric mkefiboot script with a sudo centric one
RUN echo "bash -C /usr/local/bin/finishSetup.sh" >> /home/$MYUNAME/.bashrc && \
echo "export PATH=/usr/local/bin:/localdisk/designer/$MYUNAME/bin:\$PATH" >> /home/$MYUNAME/.bashrc && \
chmod a+x /usr/local/bin/*
# Genrate a git configuration file in order to save an extra step
# for end users, this file is required by "repo" tool.
RUN chown $MYUNAME /home/$MYUNAME && \
runuser -u $MYUNAME -- git config --global user.email $MYUNAME@starlingx.com && \
runuser -u $MYUNAME -- git config --global user.name $MYUNAME && \
runuser -u $MYUNAME -- git config --global color.ui false
# Customizations for mirror creation
RUN rm /etc/yum.repos.d/CentOS-Sources.repo
RUN rm /etc/yum.repos.d/epel.repo
COPY centos-mirror-tools/yum.repos.d/* /etc/yum.repos.d/
COPY centos-mirror-tools/rpm-gpg-keys/* /etc/pki/rpm-gpg/
# Import GPG keys
RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY*
# Try to continue a yum command even if a StarlingX repo is unavailable.
RUN yum-config-manager --setopt=StarlingX\*.skip_if_unavailable=1 --save
# When we run 'init' below, it will run systemd, and systemd requires RTMIN+3
# to exit cleanly. By default, docker stop uses SIGTERM, which systemd ignores.
STOPSIGNAL RTMIN+3
# Don't know if it's possible to run services without starting this
CMD /usr/sbin/init