FusionForge

Difference between revisions of "Tools/BuildBot"

From FusionForge Wiki
Jump to: navigation, search
(Mention --no-rlimits)
(Todo/Wishlist: Document recent changes on the buildbot)
Line 208: Line 208:
 
== Todo/Wishlist ==
 
== Todo/Wishlist ==
  
* Run Debian build scripts inside the VM rather than in a cowbuilder, and unify with the Dev VM's script.
+
* Run Debian build scripts inside the VM rather than in a cowbuilder (done for master), and unify with the Dev VM's script.
 
* Use standard 'virbr0' from libvirt
 
* Use standard 'virbr0' from libvirt
* Use stock LXC templates from distro and make (documented) changes using post-install scripts?
+
* Document changes to stock LXC templates from distro (see tests/buildbot/lxc/lxc-wrapper); among other things, these changes overwrite SSH host keys and set up root's authorized_keys file)
** FusionForge LXC templates do not support newer lxc (1.0.4 -> <tt>getopt: unrecognized option '--rootfs=/var/lib/lxc/centos6.local/rootfs'</tt>)
 
** lxc-centos6 is completely broken with rinse3 (missing basic things like <tt>libblkid</tt>, <tt>iproute</tt> and starting <tt>network</tt> on boot)
 
** Note: our custom templates are responsible for overwriting the SSH host keys and copying the ~jenkins/.ssh/id_rsa.pub to ~root, so we can <tt>rsync</tt> to the host over SSH, among others.
 
 
* Automate task creation maybe using dsl plugin
 
* Automate task creation maybe using dsl plugin
 
** improve the Jenkins plugin to pilot/setup Jenkins
 
** improve the Jenkins plugin to pilot/setup Jenkins

Revision as of 12:40, 28 November 2014

buildbot.fusionforge.org hosts a Jenkins instance, handling the different build queues.


Hosting

  • Physical box: miromesnil.gnurandal.net
  • KVM VM: vladimir.gnurandal.net
  • SSH port: 10022

Host configuration

  • user: jenkins ($HOME=/var/lib/jenkins)
  • ~jenkins/jobs contains the different job configurations; for instance ~jenkins/jobs/fusionforge-master-src-debian8/config.xml runs fusionforge-build-and-test-src-deb.sh
  • pipeline plugin.

Pre-installation

Common installation setup (note: using libvirt for simplicity, while live buildbot uses some tuned dhcpd conf):

# Prepare some space
# - /var/lib/lxc/ : 1GB
# - /var/cache/lxc/ : 1GB
# - /var/lib/jenkins/ : 1GB

# LXC
echo "cgroup  /sys/fs/cgroup  cgroup  defaults  0   0" >> /etc/fstab
mount /sys/fs/cgroup

# Create user
useradd jenkins -m -d /var/lib/jenkins -s /bin/bash

# Grab the code for the installation script
apt-get install git
su - jenkins
git clone git://fusionforge.org/deb-packaging/deb-packaging.git fusionforge 
# or git clone git://fusionforge.org/fusionforge/fusionforge.git
# back to user root
exit

# VM tools and templates
cd ~jenkins/fusionforge/tests/lxc/
make

# VMs networking
apt-get install avahi-daemon libnss-mdns
sed -i -e 's/^hosts:/hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4/' /etc/nsswitch.conf
# and make sure you accept mdns traffic: iptables -A INPUT -i virbr0 -p udp --dport 5353 -j ACCEPT
apt-get install libvirt-bin dnsmasq
service dnsmasq stop
update-rc.d dnsmasq remove
sed -i -e 's/virbr0/br0/' /etc/libvirt/qemu/networks/default.xml  # compat with FusionForge LXC configs
virsh net-autostart default
service libvirt-bin restart

# Prepare sudo access
apt-get install sudo
cat <<EOF > /etc/sudoers.d/ci
Defaults env_keep += "HOME"
jenkins ALL= NOPASSWD: /usr/sbin/cowbuilder *
EOF
mkdir ~jenkins/build/ ~jenkins/reports/
chown -R jenkins: ~jenkins/

# Start/stop VMs
cat <<EOF >> /etc/sudoers.d/ci
jenkins ALL= NOPASSWD: /usr/bin/lxc-create
jenkins ALL= NOPASSWD: /usr/bin/lxc-start
jenkins ALL= NOPASSWD: /usr/bin/lxc-stop
jenkins ALL= NOPASSWD: /usr/bin/lxc-destroy
EOF

Debian setup - let's follow what fusionforge-build-and-test-src-deb.sh does:

su - jenkins

# - Prepare cowbuilder root
mkdir -p ~/builder/cow/
mkdir -p ~/builder/buildplace/
mkdir -p ~/builder/result/
cd fusionforge/
DISTROLIST=wheezy tests/scripts/manage-cowbuilder.sh

# - Sign packages
hostname -f  # < make sure hostname properly configured
gpg --batch --gen-key <<EOF
     Key-Type: RSA
     Key-Length: 2048
     Subkey-Type: RSA
     Subkey-Length: 2048
     Name-Real: buildbot@$(hostname -f)
     Expire-Date: 0
     %commit
EOF

# - Connect to VM through SSH
ssh-keygen
# back to user root
exit

# - Compile source package and check it for errors
apt-get install cowbuilder devscripts debhelper quilt lintian
# - Create a local repository
apt-get install reprepro

Now we can run the script manually for testing:

su - jenkins
cd fusionforge/
git checkout debian/5.3
# Does all steps mentioned above, installs and runs the testsuite
bash -x tests/scripts/fusionforge-build-and-test-deb.sh debian7.local

CentOS setup

apt-get install rinse  # > 1.8
apt-get install netmask dpkg-dev  # for lxc-centos6 (dpkg-dev for dpkg-architecture)
apt-get install createrepo
# avahi requires kernel > 3.9 (or, I supposed, a backported feature in CentOS');
# fixes 'SO_REUSEPORT failed: Protocol not available'
apt-get install -t wheezy-backports linux-image-amd64
git checkout Branch_5_3
bash -x tests/scripts/fusionforge-build-and-test-rpm.sh centos6.local

Debian 8 setup

Edit /lib/systemd/system/avahi-daemon.service and add --no-rlimits so it works in LXC:

ExecStart=/usr/sbin/avahi-daemon -s --no-rlimits

Current network configuration

This uses an historically more complex setup involving a manual bridge and ISC DHCPd with explicit DNS servers.

cat <<EOF >> /etc/network/interfaces
auto br0
iface br0 inet static
       bridge_stp off
       bridge_maxwait 5
       address 172.16.0.1
       netmask 255.255.0.0
       post-up echo 1 > /proc/sys/net/ipv4/ip_forward
       post-up iptables -t nat -A POSTROUTING -s '172.16.0.1/16' -o eth0 -j MASQUERADE
       post-up service isc-dhcp-server restart
       post-down iptables -t nat -D POSTROUTING -s '172.16.0.1/16' -o eth0 -j MASQUERADE
EOF

apt-get install isc-dhcp-server
sed -i -s 's/^INTERFACES=.*/INTERFACES="br0"/' /etc/default/isc-dhcp-server
cat <<EOF >> /etc/dhcp/dhcpd.conf
subnet 172.16.0.0 netmask 255.255.0.0 {
       range 172.16.0.10 172.16.0.50;
       option routers 172.16.0.1;
       option domain-name "local";
       option domain-name-servers 88.191.254.60, 88.191.254.70;
}
EOF

LXC templates

The network configuration is in e.g. tests/lxc/config.debian7:

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0

When you update one of our template scripts, remove the cached rootfs:

rm -rf /var/cache/lxc/debian-7/rootfs-wheezy-amd64/

Builds

Intended build order: http://buildbot.fusionforge.org/view/Pipeline%20for%205.3/

unittests
  → src build on centos5 → src build on centos6
  → src build on debian7 → src build on debian8
  • The scripts reinstall FusionForge from scratch in a LXC container
  • src: a 'source install' using install-ng
  • deb/rpm: a pkg build + a pkg install
  • at the end of each build, units and selenium tests are executed
  • if the build is ok, the result is put in a repository

Other tasks:

  • Stand-alone LXC templates build
  • 3rd party software

Jenkins configuration

  • Manage Jenkins > Plugins > Advanced > Check Now (update plugins list)
  • Available > "Git Plugin" > Download and Install after restart

Sample build configuration

Example with fusionforge-53-src-debian7:

In Jenkins, create a new job:

tests/scripts/fusionforge-build-and-test-deb.sh debian7.local
  • Save

Jenkins currently checks out the repository in /var/lib/jenkins/jobs/fusionforge-53-deb-debian7/workspace.

Todo/Wishlist

  • Run Debian build scripts inside the VM rather than in a cowbuilder (done for master), and unify with the Dev VM's script.
  • Use standard 'virbr0' from libvirt
  • Document changes to stock LXC templates from distro (see tests/buildbot/lxc/lxc-wrapper); among other things, these changes overwrite SSH host keys and set up root's authorized_keys file)
  • Automate task creation maybe using dsl plugin
    • improve the Jenkins plugin to pilot/setup Jenkins
    • put/visualize result in FRS
  • Use master/slave Jenkins capability to enable parallel builds
  • Try Buildbot as a Jenkins alternative
  • Use the tests/init-jenkins.sh script in the install procedure

See also