FusionForge

Difference between revisions of "Tools/BuildBot"

From FusionForge Wiki
Jump to: navigation, search
(Updated with current info)
(Todo/Wishlist)
 
(50 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Jenkins interface : http://buildbot.fusionforge.org/ (gives a list of different build queues)
+
[http://buildbot.fusionforge.org/ buildbot.fusionforge.org] hosts a Jenkins instance, handling the different build queues.
  
Configuration of the builds (TODO: check me):
 
* physical box: miromesnil.gnurandal.net
 
* buildbot VM (using KVM): vladimir.gnurandal.net
 
* ssh port : 10022
 
  
In the VM :
+
== Hosting ==
* user jenkins ($HOME=/var/lib/jenkins)
 
* ~jenkins/jobs contains the different job configurations
 
* for instance ~jenkins/jobs/fusionforge-master-src-debian8/config.xml runs https://fusionforge.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=fusionforge/fusionforge.git;a=blob;f=tests/scripts/fusionforge-build-and-test-src-deb.sh;hb=HEAD
 
  
We also use [[LXC]], using a bridge setup with DHCP.
+
* Physical box: miromesnil.gnurandal.net
 +
* KVM VM: vladimir.gnurandal.net
 +
* SSH port: 10022
  
It's rather using standard Jenkins setup, with the pipeline plugin.
+
== Host configuration ==
There are :
 
* tasks to build lxc templates
 
* 3rd party software
 
  
It is meant to have the following build order:
+
* user: jenkins (<tt>$HOME=/var/lib/jenkins</tt>)
 +
* <tt>~jenkins/jobs</tt> contains the different job configurations; for instance <tt>~jenkins/jobs/fusionforge-master-src-debian8/config.xml</tt> runs [https://fusionforge.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=fusionforge/fusionforge.git;a=blob;f=tests/scripts/fusionforge-build-and-test-src-deb.sh;hb=HEAD fusionforge-build-and-test-src-deb.sh]
  
unittests
+
=== Pre-installation ===
  → src build on centos5 → src build on centos6
 
  → src build on debian7 → src build on debian8
 
  
* build in 'src' case: a 'source install' using install-ng
+
Common installation setup (note: using libvirt for simplicity, while live buildbot uses some tuned dhcpd conf):
* build in 'pkg' case: a pkg build + a pkg install
+
<nowiki>
* at the end of each build, Selenium tests are executed
+
# Prepare some space
* if the build is ok, the result is put in a repository
+
# - /var/lib/lxc/ : 1GB
 +
# - /var/cache/lxc/ : 1GB
 +
# - /var/lib/jenkins/ : 1GB
  
TODO/wish list:
+
# Create user
 +
useradd jenkins -m -d /var/lib/jenkins -s /bin/bash
 +
 
 +
# Grab the code for the VM tools and wrapper
 +
apt-get install git
 +
git clone git://fusionforge.org/fusionforge/fusionforge.git
 +
apt-get install make
 +
make -C fusionforge/tests/buildbot/
 +
apt-get install createrepo  # for push-packages-to-repositories.sh
 +
 
 +
# VMs networking
 +
apt-get install lxc debootstrap
 +
apt-get install avahi-daemon libnss-mdns
 +
sed -i.bak -e 's/^hosts:.*/hosts:          files mdns4_minimal [NOTFOUND=return] dns/' /etc/nsswitch.conf
 +
# and make sure you accept mdns traffic: iptables -A INPUT -i virbr0 -p udp --dport 5353 -j ACCEPT
 +
apt-get install libvirt-bin dnsmasq ebtables
 +
service dnsmasq stop
 +
update-rc.d dnsmasq remove
 +
virsh net-autostart default
 +
service libvirtd restart
 +
cat > /etc/lxc/default.conf <<'EOF'
 +
lxc.network.type = veth
 +
lxc.network.flags = up
 +
lxc.network.link = virbr0
 +
EOF
 +
 
 +
# Prepare bot workspace and sudo access
 +
apt-get install sudo
 +
mkdir ~jenkins/reports/
 +
chown -R jenkins: ~jenkins/
 +
(cd fusionforge/tests/buildbot/ && ./init-jenkins.sh)
 +
# possibly add your SSH public key in ~jenkins/.ssh/id_rsa.pub to access the VMs
 +
 
 +
# possibly add your configuration, e.g.:
 +
(cd fusionforge/tests/buildbot/config/ && sed 's/^KEEPVM=.*/KEEPVM=true/' default > $(hostname))
 +
 
 +
# SSH host keys cache
 +
mkdir -m 700 /var/cache/lxc/ssh/
 +
mkdir -m 700 /var/cache/lxc/ssh/debian8.local/
 +
mkdir -m 700 /var/cache/lxc/ssh/centos7.local/
 +
cp -a .../existing_vm/rootfs/etc/ssh/ssh_host_* /var/cache/lxc/ssh/debian8.local/
 +
cp -a .../existing_vm/rootfs/etc/ssh/ssh_host_* /var/cache/lxc/ssh/centos7.local/
 +
...
 +
</nowiki>
 +
 
 +
==== Debian additional setup for 5.3 ====
 +
<nowiki>
 +
# - Compile source package outside of the container
 +
apt-get install cowbuilder
 +
apt-get install devscripts debhelper quilt lintian
 +
# - Create a local repository
 +
apt-get install reprepro
 +
# Make sure hostname properly configured
 +
hostname -f 
 +
 
 +
# Add sudo support for pdebuild and cowbuilder
 +
cat <<EOF >> /etc/sudoers.d/ci
 +
Defaults env_keep += "HOME"
 +
jenkins ALL= NOPASSWD: /usr/sbin/cowbuilder
 +
EOF
 +
 
 +
su - jenkins
 +
 
 +
# - Prepare cowbuilder root
 +
mkdir -p ~/builder/cow/
 +
mkdir -p ~/builder/buildplace/
 +
mkdir -p ~/builder/result/
 +
cd fusionforge/
 +
DISTROLIST=wheezy tests/scripts/manage-cowbuilder.sh
 +
 
 +
exit
 +
</nowiki>
 +
 
 +
==== Manual test ====
 +
 
 +
Now we can run the script manually for testing:
 +
su - jenkins
 +
* FusionForge master:
 +
(cd ~/fusionforge/ && bash -ex ./tests/buildbot/fusionforge-build-and-test.sh debian8.local deb; echo $?)
 +
(cd ~/fusionforge/ && bash -ex ./tests/buildbot/fusionforge-build-and-test.sh centos7.local rpm; echo $?)
 +
sudo /usr/local/sbin/lxc-wrapper destroy centos7
 +
(cd ~/fusionforge/ && bash -ex ./tests/buildbot/fusionforge-build-and-test.sh centos7.local src; echo $?)
 +
 
 +
=== Current network configuration at buildbot.fusionforge.org ===
 +
 
 +
It uses an historically more complex setup involving a manual bridge and ISC DHCPd with explicit DNS servers.
 +
 
 +
<nowiki>
 +
cat <<EOF >> /etc/network/interfaces
 +
auto br0
 +
iface br0 inet static
 +
      bridge_stp off
 +
      bridge_maxwait 5
 +
      address 172.16.0.1
 +
      netmask 255.255.0.0
 +
      post-up echo 1 > /proc/sys/net/ipv4/ip_forward
 +
      post-up iptables -t nat -A POSTROUTING -s '172.16.0.1/16' -o eth0 -j MASQUERADE
 +
      post-up service isc-dhcp-server restart
 +
      post-down iptables -t nat -D POSTROUTING -s '172.16.0.1/16' -o eth0 -j MASQUERADE
 +
EOF
 +
 
 +
apt-get install isc-dhcp-server
 +
sed -i -s 's/^INTERFACES=.*/INTERFACES="br0"/' /etc/default/isc-dhcp-server
 +
cat <<EOF >> /etc/dhcp/dhcpd.conf
 +
subnet 172.16.0.0 netmask 255.255.0.0 {
 +
      range 172.16.0.10 172.16.0.50;
 +
      option routers 172.16.0.1;
 +
      option domain-name "local";
 +
      option domain-name-servers 88.191.254.60, 88.191.254.70;
 +
}
 +
EOF
 +
</nowiki>
 +
 
 +
=== LXC templates ===
 +
 
 +
When you update one of our template scripts, remove the cached rootfs, e.g.:
 +
/usr/local/sbin/lxc-wrapper emptycache centos7
 +
 
 +
== Jenkins configuration ==
 +
 
 +
* Manage Jenkins > Plugins > Advanced > Check Now (update plugins list)
 +
* Available > "Git Plugin" > Download and Install after restart
 +
 
 +
== Sample build configuration ==
 +
 
 +
Example with <tt>fusionforge-60-src-debian8</tt>:
 +
 
 +
In Jenkins, create a new job:
 +
* free-style project
 +
* name: fusionforge-60-src-debian8
 +
* source code: Git: git://fusionforge.org/fusionforge/fusionforge.git
 +
* source browser: https://fusionforge.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=fusionforge/fusionforge.git
 +
* add a build step:
 +
./tests/buildbot/fusionforge-func_tests.sh debian8.local src
 +
* Save
 +
 
 +
Jenkins currently checks out the repository in <tt>/var/lib/jenkins/jobs/fusionforge-60-src-debian8/workspace</tt>.
 +
 
 +
We also upgrade the VMs weekly:
 +
* free-style project
 +
* name update-lxc-vms
 +
* build periodically: @weekly
 +
* source code: Git: git://fusionforge.org/fusionforge/fusionforge.git
 +
* add a build step:
 +
<pre>
 +
os=debian8
 +
 
 +
cd $WORKSPACE/tests
 +
sudo /usr/local/sbin/lxc-wrapper stop $os || true
 +
 
 +
sudo /usr/local/sbin/lxc-wrapper emptycache $os
 +
sudo /usr/local/sbin/lxc-wrapper prepare $os
 +
 
 +
# Not sure about re-introducing common-vm and ./start_vm just for this:
 +
#./buildbot/start_vm $os.local
 +
sudo /usr/local/sbin/lxc-wrapper stop $os || true
 +
</pre>
 +
 
 +
== Todo/Wishlist ==
 +
 
 +
* Document changes to stock LXC templates from distro (see tests/buildbot/lxc/lxc-wrapper); among other things, these changes overwrite SSH host keys and set up root's authorized_keys file)
 
* Automate task creation maybe using dsl plugin
 
* Automate task creation maybe using dsl plugin
 
** improve the Jenkins plugin to pilot/setup Jenkins
 
** improve the Jenkins plugin to pilot/setup Jenkins
 
** put/visualize result in FRS
 
** put/visualize result in FRS
 +
* Use standard 'virbr0' from libvirt also at vladimir
 
* Use master/slave Jenkins capability to enable parallel builds
 
* Use master/slave Jenkins capability to enable parallel builds
 
* Try Buildbot as a Jenkins alternative
 
* Try Buildbot as a Jenkins alternative
 +
* document ~/.mini-dinstall.conf which is used for deb repo upload.
  
 
== See also ==
 
== See also ==
Line 40: Line 195:
 
* [[Virtual machine development environment]]: run the test suite in a complete preconfigured environment
 
* [[Virtual machine development environment]]: run the test suite in a complete preconfigured environment
 
* [[Test_Suite]]: what tests are run
 
* [[Test_Suite]]: what tests are run
 +
* [http://jenkins-ci.org/ Jenkins homepage]
 +
* [https://wiki.debian.org/LXC LXC] page on the Debian wiki
  
 
[[Category:Development]]
 
[[Category:Development]]

Latest revision as of 17:39, 4 October 2017

buildbot.fusionforge.org hosts a Jenkins instance, handling the different build queues.


Hosting

  • Physical box: miromesnil.gnurandal.net
  • KVM VM: vladimir.gnurandal.net
  • SSH port: 10022

Host configuration

  • user: jenkins ($HOME=/var/lib/jenkins)
  • ~jenkins/jobs contains the different job configurations; for instance ~jenkins/jobs/fusionforge-master-src-debian8/config.xml runs fusionforge-build-and-test-src-deb.sh

Pre-installation

Common installation setup (note: using libvirt for simplicity, while live buildbot uses some tuned dhcpd conf):

# Prepare some space
# - /var/lib/lxc/ : 1GB
# - /var/cache/lxc/ : 1GB
# - /var/lib/jenkins/ : 1GB

# Create user
useradd jenkins -m -d /var/lib/jenkins -s /bin/bash

# Grab the code for the VM tools and wrapper
apt-get install git
git clone git://fusionforge.org/fusionforge/fusionforge.git
apt-get install make
make -C fusionforge/tests/buildbot/
apt-get install createrepo  # for push-packages-to-repositories.sh

# VMs networking
apt-get install lxc debootstrap
apt-get install avahi-daemon libnss-mdns
sed -i.bak -e 's/^hosts:.*/hosts:          files mdns4_minimal [NOTFOUND=return] dns/' /etc/nsswitch.conf
# and make sure you accept mdns traffic: iptables -A INPUT -i virbr0 -p udp --dport 5353 -j ACCEPT
apt-get install libvirt-bin dnsmasq ebtables
service dnsmasq stop
update-rc.d dnsmasq remove
virsh net-autostart default
service libvirtd restart
cat > /etc/lxc/default.conf <<'EOF'
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = virbr0
EOF

# Prepare bot workspace and sudo access
apt-get install sudo
mkdir ~jenkins/reports/
chown -R jenkins: ~jenkins/
(cd fusionforge/tests/buildbot/ && ./init-jenkins.sh)
# possibly add your SSH public key in ~jenkins/.ssh/id_rsa.pub to access the VMs

# possibly add your configuration, e.g.:
(cd fusionforge/tests/buildbot/config/ && sed 's/^KEEPVM=.*/KEEPVM=true/' default > $(hostname))

# SSH host keys cache
mkdir -m 700 /var/cache/lxc/ssh/
mkdir -m 700 /var/cache/lxc/ssh/debian8.local/
mkdir -m 700 /var/cache/lxc/ssh/centos7.local/
cp -a .../existing_vm/rootfs/etc/ssh/ssh_host_* /var/cache/lxc/ssh/debian8.local/
cp -a .../existing_vm/rootfs/etc/ssh/ssh_host_* /var/cache/lxc/ssh/centos7.local/
...

Debian additional setup for 5.3

# - Compile source package outside of the container
apt-get install cowbuilder
apt-get install devscripts debhelper quilt lintian
# - Create a local repository
apt-get install reprepro
# Make sure hostname properly configured
hostname -f  

# Add sudo support for pdebuild and cowbuilder
cat <<EOF >> /etc/sudoers.d/ci
Defaults env_keep += "HOME"
jenkins ALL= NOPASSWD: /usr/sbin/cowbuilder
EOF

su - jenkins

# - Prepare cowbuilder root
mkdir -p ~/builder/cow/
mkdir -p ~/builder/buildplace/
mkdir -p ~/builder/result/
cd fusionforge/
DISTROLIST=wheezy tests/scripts/manage-cowbuilder.sh

exit

Manual test

Now we can run the script manually for testing:

su - jenkins
  • FusionForge master:
(cd ~/fusionforge/ && bash -ex ./tests/buildbot/fusionforge-build-and-test.sh debian8.local deb; echo $?)
(cd ~/fusionforge/ && bash -ex ./tests/buildbot/fusionforge-build-and-test.sh centos7.local rpm; echo $?)
sudo /usr/local/sbin/lxc-wrapper destroy centos7
(cd ~/fusionforge/ && bash -ex ./tests/buildbot/fusionforge-build-and-test.sh centos7.local src; echo $?)

Current network configuration at buildbot.fusionforge.org

It uses an historically more complex setup involving a manual bridge and ISC DHCPd with explicit DNS servers.

cat <<EOF >> /etc/network/interfaces
auto br0
iface br0 inet static
       bridge_stp off
       bridge_maxwait 5
       address 172.16.0.1
       netmask 255.255.0.0
       post-up echo 1 > /proc/sys/net/ipv4/ip_forward
       post-up iptables -t nat -A POSTROUTING -s '172.16.0.1/16' -o eth0 -j MASQUERADE
       post-up service isc-dhcp-server restart
       post-down iptables -t nat -D POSTROUTING -s '172.16.0.1/16' -o eth0 -j MASQUERADE
EOF

apt-get install isc-dhcp-server
sed -i -s 's/^INTERFACES=.*/INTERFACES="br0"/' /etc/default/isc-dhcp-server
cat <<EOF >> /etc/dhcp/dhcpd.conf
subnet 172.16.0.0 netmask 255.255.0.0 {
       range 172.16.0.10 172.16.0.50;
       option routers 172.16.0.1;
       option domain-name "local";
       option domain-name-servers 88.191.254.60, 88.191.254.70;
}
EOF

LXC templates

When you update one of our template scripts, remove the cached rootfs, e.g.:

/usr/local/sbin/lxc-wrapper emptycache centos7

Jenkins configuration

  • Manage Jenkins > Plugins > Advanced > Check Now (update plugins list)
  • Available > "Git Plugin" > Download and Install after restart

Sample build configuration

Example with fusionforge-60-src-debian8:

In Jenkins, create a new job:

./tests/buildbot/fusionforge-func_tests.sh debian8.local src
  • Save

Jenkins currently checks out the repository in /var/lib/jenkins/jobs/fusionforge-60-src-debian8/workspace.

We also upgrade the VMs weekly:

os=debian8

cd $WORKSPACE/tests
sudo /usr/local/sbin/lxc-wrapper stop $os || true

sudo /usr/local/sbin/lxc-wrapper emptycache $os
sudo /usr/local/sbin/lxc-wrapper prepare $os

# Not sure about re-introducing common-vm and ./start_vm just for this:
#./buildbot/start_vm $os.local
sudo /usr/local/sbin/lxc-wrapper stop $os || true

Todo/Wishlist

  • Document changes to stock LXC templates from distro (see tests/buildbot/lxc/lxc-wrapper); among other things, these changes overwrite SSH host keys and set up root's authorized_keys file)
  • Automate task creation maybe using dsl plugin
    • improve the Jenkins plugin to pilot/setup Jenkins
    • put/visualize result in FRS
  • Use standard 'virbr0' from libvirt also at vladimir
  • Use master/slave Jenkins capability to enable parallel builds
  • Try Buildbot as a Jenkins alternative
  • document ~/.mini-dinstall.conf which is used for deb repo upload.

See also