FusionForge

Difference between revisions of "Virtual machine development environment"

From FusionForge Wiki
Jump to: navigation, search
m
 
(127 intermediate revisions by 5 users not shown)
Line 1: Line 1:
Roland Mas has designed a virtual machine image (aka [[FusionForge Sandbox]]) to provide a development / test machine providing an already configured automated tests environment.
 
  
See : http://lists.fusionforge.org/pipermail/fusionforge-general/2010-November/001245.html for more details
+
If you are looking for the root password for the .ova file ,
 +
(the other is a .vagrant file, contrary to what the posting says) :
  
Here's an excerpt from the enclosed <tt>README.html</tt> :
+
http://lists.fusionforge.org/pipermail/fusionforge-general/2010-November/001245.html
----
 
<tt>
 
'''FusionForge sandbox'''
 
  
Welcome to the FusionForge sandbox. This VM/appliance is designed to help you get started on development. It's basically a clean Debian system (user:pass is root:root),
+
http://fusionforge.fusionforge.org/sandbox/
with a checkout of the FusionForge code and a few helper scripts. Those scripts are in the '''/root/scripts''' directory:
 
  
  • '''update.sh''' will update the sources from FusionForge's Subversion repository, and also the currently installed packages.
+
it is user: root
  • '''build.sh''' builds FusionForge packages from the sources, and stores them in the package repository at /root/debian-repository. (rem Cbac starting from a debboostraped vm, needs to add dupload, bzr-svn and svn).
 
  • '''install.sh''' installs those FusionForge packages. This should end up with an up-to-date FusionForge running on http://forge.local/.
 
  • '''start-selenium.sh''' gets and runs Selenium, which is needed for the testsuite.
 
  • '''run-testsuite.sh''' actually starts the testsuite.
 
  • '''install-gui.sh''' sets up an X11 graphical environment.
 
  
The actual source code is stored under /root/fusionforge-trunk, so that's where you should do the changes you want to test.
+
password: vagrant
  
'''Running everything on the VM'''
+
the .ova gives you a root-only XFCE running FF locally with scripts under  /usr/src/fusion...  .
  
If you run install-gui.sh, then you get a graphical desktop environment on the VM. To build and install from sources, run the “Build + inst”
 
script (there's an icon on the desktop). You can then use http://forge.local/, with the administrator's login being “admin” and its password being
 
“myadmin”. To run the [[Test Suite]], launch Selenium and keep it running (possibly iconized) while you launch the “Run testsuite”.
 
  
'''Running remotely'''
 
  
If you don't want to install a full GUI on the VM, you can also run the interesting things remotely through SSH. Depending on the configuration of
+
We provide a 64-bit development/test Virtual Machine (aka FusionForge Sandbox) with an already configured automated tests environment. FusionForge is not pre-installed, it is packaged and installed using included automated scripts.
your VM, the network settings may change, but the VM displays its IP address at boot time, so you can ssh root@192.168.x.x and work in there. Note
 
that Selenium needs an X11 display (since it drives a graphical web browser), so you'll need to run ssh -X root@192.168.x.x start-selenium.sh
 
(note the -X).
 
</tt>
 
----
 
  
You may reuse the pre-configured VM image with Virtualbox (or any other VM system, converting the image to the appropriate format), which is the easiest path, or setup a development environment on a Debian squeeze system you installed on your own, by using the <tt>VM-scripts/configure-scripts.sh</tt> script (see below).
+
== Dependencies : Vagrant and VirtualBox ==
  
== Tips ==
+
To use the VM, please [http://docs.vagrantup.com/v2/installation/ install Vagrant] (>= v1.1).
  
* To reset the admin password, use : <tt>/usr/share/gforge/bin/forge_set_password admin</tt>
+
By default Vagrant uses [https://www.virtualbox.org/wiki/Downloads VirtualBox] so install it too.
  
* Converting from Virtualbox image format to qemu QCOW2 : convert to raw format first
+
* If you're running a recent Debian or Ubuntu, the following should work:
  $ VBoxManage clonehd -format RAW ffsandbox.vmdk ffsandbox.raw
+
aptitude install vagrant virtualbox
 +
Note: VirtualBox recently moved to the <tt>contrib</tt> section, so you need to enable it in your <tt>/etc/apt/sources.list</tt>, for instance by adding something like :
 +
  deb http://ftp.fr.debian.org/debian/ jessie main contrib
  
Then convert to QCOW2 format :
+
Make sure that you enabled '''hardware virtualization''' in your computer's BIOS, otherwise VirtualBox may not be able to run the VM.
$ qemu-img convert -f raw -O qcow2 ffsandbox.raw ffsandbox.qcow2
 
  
* Running inside qemu :
+
See also [[#Using_QEMU.2FKVM_instead_of_VirtualBox|use it with KVM]] if you want to extract the disk image from the <tt>.box</tt>.
** Create a VM by importing the created qcow2 disk image, with <tt>virt-manager</tt>.
 
** You may need to set manually the driver type to 'qcow2' in "disk" device section in the VM xml file (in /etc/libvirt/qemu/). In this case you need to redefine the VM xml file by: `sudo virsh -c qemu:///system define /etc/libvirt/qemu/ffsandbox.xml` then start the VM.
 
** You need to reconfigure network interfaces in <tt>/etc/network/interfaces</tt> afterwards for replicating <tt>eth0</tt> config stanza for <tt>eth1</tt> for instance. You can find which device to enable by running ifconfig -a.
 
** You may need to check the <tt>web_host</tt> setting in <tt>/etc/fusionforge/config.ini.d/debian-install.ini</tt> to adjust the forge's URLs for redirections.
 
  
* It may not harm using <tt>aptitude</tt> inside the VM to upgrade at first launch, eventually taking this oportunity to configure the proper keymap and associate stuff.
+
== Starting the VM with Vagrant ==
  
* Avoiding firefox self-signed certificate error dialogs when running selenium tests :
+
Note: installing the VM requires some free space (~700MB in your <tt>~/.vagrant.d/</tt> and ~1.5GB in <tt>~/VirtualBox VMs</tt>).
** Start connection with valid DISPLAY (ssh -X)
 
** Configure a new firefox profile in a directory of your choice (for instance : <tt>/root/fusionforge-trunk/tests/ff-profile-selenium/</tt>) with :
 
# firefox -ProfileManager
 
** Still in firefox, visit https://forge.local/ and add an exception for the self-signed certificate
 
** Run the tests with the corresponding profile (change the <tt>run:</tt> rule of <tt>/root/fusionforge-trunk/3rd-party/selenium/Makefile</tt>) as :
 
  cd binary/selenium-server-1.0.3 ; LANG=C java -jar selenium-server.jar -firefoxProfileTemplate /root/fusionforge-trunk/tests/ff-profile-selenium/ -interactive
 
** restart <tt>start-selenium.sh</tt>
 
  
* You may prefer running the X display inside the VM to start the selenium server (run with ./start-selenium.sh), so that windows opened by the tests don't bother you, rather than through a ssh -X made to the VM.
+
* Save this [https://scm.fusionforge.org/anonscm/gitweb/?p=fusionforge/fusionforge.git;a=blob_plain;f=vm/Vagrantfile-sample;hb=6.0 Vagrant configuration] in a directory of your choice and rename it as <tt>Vagrantfile</tt>.
 +
* Edit the Vagrantfile if needed (e.g. if <tt>192.168.35.11</tt> is already taken on your network)
 +
* In your host's <tt>/etc/hosts</tt>, add:
 +
192.168.35.11 forge.internal lists.forge.internal scm.forge.internal users.forge.internal
 +
* Go in the directory where the Vagrantfile was saved
 +
cd {the directory you have chosen}
 +
* Download, create and start the VM with:
 +
vagrant up
 +
* You now can login without a password:
 +
vagrant ssh -- -l root
  
* running the latest version of the scripts to test the 5.1 branch may be as easy as :
+
If the VM doesn't start, try to launch the VirtualBox manager and start the VM manually: you'll have a more detailed error message.
  # wget -q -O - "https://fusionforge.org/scm/viewvc.php/*checkout*/branches/Branch_5_1/tools/VM-scripts/configure-scripts.sh?root=fusionforge" | sh -s Branch_5_1
+
 
 +
Note: for an alternative with a NAT, cf. the tip [[#Accessing_the_web_interface_from_host_-_alternative_with_port_redirection|alternative with port redirection]] below.
 +
 
 +
== Building and installing FusionForge ==
 +
 
 +
This VM/appliance is basically a clean Debian system (jessie/stable), with a git clone/checkout of the FusionForge [https://scm.fusionforge.org/anonscm/gitweb/?p=fusionforge/fusionforge.git;a=shortlog;h=6.0 debian/6.0] code in <tt>/usr/src/fusionforge/</tt>. Make sure to update with:
 +
cd /usr/src/fusionforge/
 +
git pull
 +
apt-get update
 +
 
 +
To build and install FusionForge, you just need to run helper scripts from the <tt>/usr/src/fusionforge/autoinstall/</tt> directory, namely:
 +
* '''<tt>build.sh</tt>''' automatically builds FusionForge Debian (or RPM) packages from source, and stores them in a local package repository in <tt>/usr/src/debian-repository/</tt>.
 +
* '''<tt>install.sh</tt>''' installs these FusionForge packages.
 +
 
 +
You now have an up-to-date FusionForge running on http://forge.internal/  , but only inside of the VM. So you need to install the GUI using the script install-GUI.sh
 +
The fusionforge administrator's web login is <tt>admin</tt>, set the admin password with:
 +
forge_set_password admin 'my%admin'
 +
 
 +
== Running the test suite ==
 +
 
 +
The VM is ready to run the ''functionnal'' [[Test_Suite|test suite]], which uses PHPUnit and Selenium to drive a controlled Web browser through various tests of the different features of the forge (hence ''functionnal'' tests), using its highest-level interface: its Web interface.
 +
 
 +
''Note: you don't need to do this if you just want to test FusionForge. This is for running the test suite to ensure you didn't break anything when developping.''
 +
 
 +
A couple other scripts run the test suite:
 +
* '''<tt>tests/func_tests.sh</tt>''' installs Selenium and runs the testsuite; use TESTGLOB='func/sometest/*' to limit the tests
 +
* '''<tt>autoinstall/install-gui.sh</tt>''' optionally sets up an X11 graphical environment.
 +
 
 +
There's also an icon on the VM's Desktop to run <tt>build.sh+install.sh</tt>, and another one for <tt>func_tests.sh</tt>.
 +
 
 +
Tests need around 25mn to complete, depending on the computing power of your machine.
 +
Failure screenshots are stored in <tt>/var/log/</tt>.
 +
 
 +
=== Running graphical programs remotely ===
 +
 
 +
If you don't want to install a full GUI on the VM (i.e. not using <tt>install-gui.sh</tt>), you can also run the interesting things remotely through SSH using
 +
vagrant ssh -- -l root -X
 +
 
 +
Note that since Selenium needs an X11 display (as it drives a graphical web browser), you may then see the full graphical run of the test suite on your local X display in a new Firefox window.
 +
 
 +
You may prefer running the X display inside the VM to start the selenium server, so that windows opened by the tests don't bother you.
 +
 
 +
=== Headless run ===
 +
 
 +
'''<tt>func_tests-xvnc.sh</tt>''' runs the testsuite after setting up a fake VNC/X11 environment:
 +
tests/func_tests-xvnc.sh deb/debian
 +
 
 +
=== Run only some tests ===
 +
 
 +
Use the TESTGLOB environment variable, e.g.:
 +
TESTGLOB='func/50_PluginsScmGit/*' ./func_tests.sh deb/debian
 +
 
 +
Another example in headless mode:
 +
TESTGLOB='func/30_RBAC/*' ./func_tests-xvnc.sh src/debian
 +
 
 +
== Tips, Issues and FAQs ==
 +
 
 +
=== To reset the FusionForge admin password ===
 +
 
 +
forge_set_password admin yournewpassword
 +
 
 +
FF deliberately is so terse. So you don't know whether its a web passwd or a bash login passwd. very amateurish.
 +
 
 +
=== Accessing the web interface from host - alternative with port redirection ===
 +
 
 +
A bit more complicated than using an internal IP, but doesn't need a separate IP.
 +
 
 +
In the <tt>Vagrantfile</tt>, add:
 +
config.vm.network :forwarded_port, guest: 80,  host: 8080
 +
config.vm.network :forwarded_port, guest: 443, host: 8443
 +
 
 +
In <tt>/etc/fusionforge/config.ini.d/zzzz-local.ini</tt>, add:
 +
[core]
 +
http_port=8080
 +
https_port=8443
 +
 
 +
And in your host's <tt>/etc/hosts</tt>, add:
 +
127.0.0.1 forge.internal lists.forge.internal scm.forge.internal users.forge.internal
 +
 
 +
Then hit http://forge.internal:8080/ !
 +
 
 +
Attention : the test suite might not work well in such a configuration
 +
 
 +
=== Using psql as root ===
 +
 
 +
su - postgres -c "psql -c 'CREATE ROLE root WITH LOGIN SUPERUSER'"
 +
 
 +
=== Use the development Git branch ===
 +
 
 +
If you want to use the latest development branch (to contribute to FusionForge or test the unreleased features), you can:
 +
cd /usr/src/fusionforge/
 +
git checkout debian/master
 +
 
 +
Note: if you switch back to an earlier version (e.g. from development back to stable), make sure you uninstall your FusionForge first because the packages won't be downgraded automatically:
 +
autoinstall/uninstall.sh
 +
rm -rf /usr/src/debian-repository/
 +
 
 +
=== Block outgoing e-mails ===
 +
 
 +
Useful if you're working on a database import with real user e-mails...
 +
iptables -A OUTPUT -p tcp --dport 25 -j REJECT --reject-with tcp-reset
 +
 
 +
== Using QEMU/KVM instead of VirtualBox ==
 +
 
 +
=== Converting from Virtualbox image format to QEMU qcow2 ===
 +
 
 +
Converting from Virtualbox image format to QEMU qcow2: convert to raw format first
 +
tar xf fusionforge-dev.box fusionforge-dev-disk1.vmdk
 +
VBoxManage clonehd -format RAW fusionforge-dev-disk1.vmdk ffsandbox.raw # ~10GB
 +
 
 +
Then optionaly convert to QCOW2 format:
 +
qemu-img convert -f raw -O qcow2 ffsandbox.raw ffsandbox.qcow2
 +
 
 +
=== Running inside libvirt ===
 +
 
 +
* Create a VM by importing the created qcow2 disk image, with <tt>virt-manager</tt>.
 +
* You may need to set manually the driver type to 'qcow2' in "disk" device section in the VM xml file (in /etc/libvirt/qemu/). In this case you need to redefine the VM xml file by: `sudo virsh -c qemu:///system define /etc/libvirt/qemu/ffsandbox.xml` then start the VM.
 +
* You need to reconfigure network interfaces in <tt>/etc/network/interfaces</tt> afterwards for replicating <tt>eth0</tt> config stanza for <tt>eth1</tt> for instance. You can find which device to enable by running <tt>ifconfig -a</tt>.
 +
* You may need to check the <tt>web_host</tt> setting in <tt>/etc/fusionforge/config.ini.d/debian-install.ini</tt> to adjust the forge's URLs for redirections.
 +
 
 +
== CentOS variant ==
 +
 
 +
A version of the VM for CentOS 7 is available:
 +
http://fusionforge.fusionforge.org/sandbox/fusionforge-dev-centos7.box
 +
 
 +
The above scripts also work for this VM.
 +
 
 +
Properly set your hostname though:
 +
echo '192.168.35.11 forge.internal lists.forge.internal scm.forge.internal users.forge.internal' >> /etc/hosts
 +
service nscd restart
 +
 
 +
== Modify the VM ==
 +
 
 +
The VM is automatically generated from scratch using the [http://www.packer.io/ Packer] tool.
 +
See the Packer configuration in <tt>vm/packer/</tt>. One advantage of packer is its ability to create a VM with a disk of arbitrary size.
 +
 
 +
== LXC / buildbot development environment ==
 +
 
 +
If you plan to test FusionForge under multiple distros and versions, you can reproduce the LXC setup used by our build bot on your computer.
 +
 
 +
It supports Debian 7 & 8 and CentOS 5, 6 & 7.
 +
 
 +
See [[Tools/BuildBot]].
 +
 
 +
== Links ==
 +
 
 +
* [http://lists.fusionforge.org/pipermail/fusionforge-general/2010-November/001245.html Original announcement]
 +
* [http://lists.fusionforge.org/pipermail/fusionforge-general/2014-January/002539.html Vagrant/Packer announcement]
 +
 
 +
[[Category:Development]]

Latest revision as of 04:44, 23 February 2018

If you are looking for the root password for the .ova file , (the other is a .vagrant file, contrary to what the posting says) :

http://lists.fusionforge.org/pipermail/fusionforge-general/2010-November/001245.html

http://fusionforge.fusionforge.org/sandbox/

it is user: root

password: vagrant

the .ova gives you a root-only XFCE running FF locally with scripts under /usr/src/fusion... .


We provide a 64-bit development/test Virtual Machine (aka FusionForge Sandbox) with an already configured automated tests environment. FusionForge is not pre-installed, it is packaged and installed using included automated scripts.

Dependencies : Vagrant and VirtualBox

To use the VM, please install Vagrant (>= v1.1).

By default Vagrant uses VirtualBox so install it too.

  • If you're running a recent Debian or Ubuntu, the following should work:
aptitude install vagrant virtualbox

Note: VirtualBox recently moved to the contrib section, so you need to enable it in your /etc/apt/sources.list, for instance by adding something like :

deb http://ftp.fr.debian.org/debian/ jessie main contrib

Make sure that you enabled hardware virtualization in your computer's BIOS, otherwise VirtualBox may not be able to run the VM.

See also use it with KVM if you want to extract the disk image from the .box.

Starting the VM with Vagrant

Note: installing the VM requires some free space (~700MB in your ~/.vagrant.d/ and ~1.5GB in ~/VirtualBox VMs).

  • Save this Vagrant configuration in a directory of your choice and rename it as Vagrantfile.
  • Edit the Vagrantfile if needed (e.g. if 192.168.35.11 is already taken on your network)
  • In your host's /etc/hosts, add:
192.168.35.11 forge.internal lists.forge.internal scm.forge.internal users.forge.internal
  • Go in the directory where the Vagrantfile was saved
cd {the directory you have chosen} 
  • Download, create and start the VM with:
vagrant up
  • You now can login without a password:
vagrant ssh -- -l root

If the VM doesn't start, try to launch the VirtualBox manager and start the VM manually: you'll have a more detailed error message.

Note: for an alternative with a NAT, cf. the tip alternative with port redirection below.

Building and installing FusionForge

This VM/appliance is basically a clean Debian system (jessie/stable), with a git clone/checkout of the FusionForge debian/6.0 code in /usr/src/fusionforge/. Make sure to update with:

cd /usr/src/fusionforge/
git pull
apt-get update

To build and install FusionForge, you just need to run helper scripts from the /usr/src/fusionforge/autoinstall/ directory, namely:

  • build.sh automatically builds FusionForge Debian (or RPM) packages from source, and stores them in a local package repository in /usr/src/debian-repository/.
  • install.sh installs these FusionForge packages.

You now have an up-to-date FusionForge running on http://forge.internal/ , but only inside of the VM. So you need to install the GUI using the script install-GUI.sh The fusionforge administrator's web login is admin, set the admin password with:

forge_set_password admin 'my%admin'

Running the test suite

The VM is ready to run the functionnal test suite, which uses PHPUnit and Selenium to drive a controlled Web browser through various tests of the different features of the forge (hence functionnal tests), using its highest-level interface: its Web interface.

Note: you don't need to do this if you just want to test FusionForge. This is for running the test suite to ensure you didn't break anything when developping.

A couple other scripts run the test suite:

  • tests/func_tests.sh installs Selenium and runs the testsuite; use TESTGLOB='func/sometest/*' to limit the tests
  • autoinstall/install-gui.sh optionally sets up an X11 graphical environment.

There's also an icon on the VM's Desktop to run build.sh+install.sh, and another one for func_tests.sh.

Tests need around 25mn to complete, depending on the computing power of your machine. Failure screenshots are stored in /var/log/.

Running graphical programs remotely

If you don't want to install a full GUI on the VM (i.e. not using install-gui.sh), you can also run the interesting things remotely through SSH using

vagrant ssh -- -l root -X

Note that since Selenium needs an X11 display (as it drives a graphical web browser), you may then see the full graphical run of the test suite on your local X display in a new Firefox window.

You may prefer running the X display inside the VM to start the selenium server, so that windows opened by the tests don't bother you.

Headless run

func_tests-xvnc.sh runs the testsuite after setting up a fake VNC/X11 environment:

tests/func_tests-xvnc.sh deb/debian

Run only some tests

Use the TESTGLOB environment variable, e.g.:

TESTGLOB='func/50_PluginsScmGit/*' ./func_tests.sh deb/debian

Another example in headless mode:

TESTGLOB='func/30_RBAC/*' ./func_tests-xvnc.sh src/debian

Tips, Issues and FAQs

To reset the FusionForge admin password

forge_set_password admin yournewpassword

FF deliberately is so terse. So you don't know whether its a web passwd or a bash login passwd. very amateurish.

Accessing the web interface from host - alternative with port redirection

A bit more complicated than using an internal IP, but doesn't need a separate IP.

In the Vagrantfile, add:

config.vm.network :forwarded_port, guest: 80,  host: 8080
config.vm.network :forwarded_port, guest: 443, host: 8443

In /etc/fusionforge/config.ini.d/zzzz-local.ini, add:

[core]
http_port=8080
https_port=8443

And in your host's /etc/hosts, add:

127.0.0.1 forge.internal lists.forge.internal scm.forge.internal users.forge.internal

Then hit http://forge.internal:8080/ !

Attention : the test suite might not work well in such a configuration

Using psql as root

su - postgres -c "psql -c 'CREATE ROLE root WITH LOGIN SUPERUSER'"

Use the development Git branch

If you want to use the latest development branch (to contribute to FusionForge or test the unreleased features), you can:

cd /usr/src/fusionforge/
git checkout debian/master

Note: if you switch back to an earlier version (e.g. from development back to stable), make sure you uninstall your FusionForge first because the packages won't be downgraded automatically:

autoinstall/uninstall.sh
rm -rf /usr/src/debian-repository/

Block outgoing e-mails

Useful if you're working on a database import with real user e-mails...

iptables -A OUTPUT -p tcp --dport 25 -j REJECT --reject-with tcp-reset

Using QEMU/KVM instead of VirtualBox

Converting from Virtualbox image format to QEMU qcow2

Converting from Virtualbox image format to QEMU qcow2: convert to raw format first

tar xf fusionforge-dev.box fusionforge-dev-disk1.vmdk
VBoxManage clonehd -format RAW fusionforge-dev-disk1.vmdk ffsandbox.raw  # ~10GB

Then optionaly convert to QCOW2 format:

qemu-img convert -f raw -O qcow2 ffsandbox.raw ffsandbox.qcow2

Running inside libvirt

  • Create a VM by importing the created qcow2 disk image, with virt-manager.
  • You may need to set manually the driver type to 'qcow2' in "disk" device section in the VM xml file (in /etc/libvirt/qemu/). In this case you need to redefine the VM xml file by: `sudo virsh -c qemu:///system define /etc/libvirt/qemu/ffsandbox.xml` then start the VM.
  • You need to reconfigure network interfaces in /etc/network/interfaces afterwards for replicating eth0 config stanza for eth1 for instance. You can find which device to enable by running ifconfig -a.
  • You may need to check the web_host setting in /etc/fusionforge/config.ini.d/debian-install.ini to adjust the forge's URLs for redirections.

CentOS variant

A version of the VM for CentOS 7 is available: http://fusionforge.fusionforge.org/sandbox/fusionforge-dev-centos7.box

The above scripts also work for this VM.

Properly set your hostname though:

echo '192.168.35.11 forge.internal lists.forge.internal scm.forge.internal users.forge.internal' >> /etc/hosts
service nscd restart

Modify the VM

The VM is automatically generated from scratch using the Packer tool. See the Packer configuration in vm/packer/. One advantage of packer is its ability to create a VM with a disk of arbitrary size.

LXC / buildbot development environment

If you plan to test FusionForge under multiple distros and versions, you can reproduce the LXC setup used by our build bot on your computer.

It supports Debian 7 & 8 and CentOS 5, 6 & 7.

See Tools/BuildBot.

Links