Veewee

What is Veewee?

Veewee is a tool for easily (and repeatedly) building custom Vagrant base boxes, KVMs, and virtual machine images.

Why Veewee?

Vagrant is a great tool for creating and configuring lightweight, reproducible, portable virtual machine environments - often used with the addition of automation tools such as Chef or Puppet.

The first step to build a new virtual machine is to download an existing 'base box'. I believe this scares a lot of people as they don't know how these unverified boxes were built. Therefore a lot of people end up building their own base box which is often time consuming and cumbersome. Veewee aims to automate all the steps for building base boxes and to collect best practices in a transparent way.

Supported VM Providers

Veewee isn't only for Vagrant. It currently supports exporting VM images for the following providers

• VirtualBox - exports to OVA/OVF filetype
• VMware (Fusion) - exports to OVA filetype
• KVM - exports to IMG filetype
• Parallels

Getting Started

Requirements

1. Virtualization Providers

2. Development Libraries

 Veewee is written in Ruby. In order to use Veewee you need Ruby installed as well as some header files in order to compile native extensions that come as dependencies. If you already have experiences with Ruby this should be very straightforward.

 For Linux

 On Linux, you may need these packages in order to build native rubygems:

• libxslt1-dev
• libxml2-dev
• zlib1g-dev # or build-essential

For Mac OS X

On Macs, either install Xcode or use homebrew to install apple-gcc42 or build-essential.

3. Ruby Environment

Use RVM or rbenv.

Installation

Install as a gem

gem install veewee

Projects that include the veewee gem can also benefit from utilizing Ruby version management (see below).

NOTE: The Veewee project is moving quickly and the Rubygem might be outdated. It is recommended that we install Veewee from source.

Install from source

In this case, use rbenv as Ruby Version Manager

$ rbenv install 1.9.3-p392
$ rbenv rehash

Clone the veewee project from source

$ cd <path_to_workspace>
$ git clone https://github.com/jedi4ever/veewee.git
$ cd veewee

Set the local ruby version within the current directory

$ rbenv local 1.9.3-p392
$ rbenv rehash

Run bundle install to install Gemfile dependencies for our selected ruby version

$ gem install bundler
$ rbenv rehash
$ bundle install
$ rbenv rehash

Command Options

Veewee with bundle exec

Unless you've created an alias for veewee, always include the bundle exec prefix like this

$ bundle exec veewee

Veewee without bundle exec

Create an alias for the veewee command so we don't have to type bundle exec any longer.

Add the alias alias veewee='bundle exec veewee' to your shell's default startup file (e.g. ~/.bash_profile)

echo "alias veewee='bundle exec veewee'" >> ~/.bash_profile

For Ubuntu: Modify ~/.profile instead of ~/.bash_profile (it should NOT present anyway, otherwise ~/.profile will not be read by Bash). 

For Zsh: Modify ~/.zshrc file instead of ~/.bash_profile.

Veewee as a Vagrant Plugin

Veewee can be used as a Vagrant plugin.

As a plugin, Veewee introduces the subcommand basebox on top of the vagrant command

$ vagrant basebox
Usage: vagrant basebox <command> [<args>]

This allows you to use the vagrant command style, which may feel more natural if you're already used to working with Vagrant. Please see Veewee's Vagrant doc for more details.

Veewee Commands

Overview of veewee commands

$ bundle exec veewee
Commands:
  veewee fusion          # Subcommand for Vmware fusion
  veewee help [COMMAND]  # Describe available commands or one specific command
  veewee kvm             # Subcommand for KVM
  veewee parallels       # Subcommand for Parallels
  veewee vbox            # Subcommand for VirtualBox
  veewee version         # Prints the Veewee version information

Veewee Provider Subcommands

Overview of veewee provider subcommands (vbox)

# Usage: bundle exec veewee <provider>
 
$ bundle exec veewee vbox
Commands:
  veewee vbox build [BOX_NAME]                     # Build box
  veewee vbox copy [BOX_NAME] [SRC] [DST]          # Copy a file to the VM
  veewee vbox define [BOX_NAME] [TEMPLATE]         # Define a new basebox starting from a template
  veewee vbox destroy [BOX_NAME]                   # Destroys the virtualmachine that was built
  veewee vbox export [BOX_NAME]                    # Exports the basebox to the vagrant format
  veewee vbox halt [BOX_NAME]                      # Activates a shutdown the virtualmachine
  veewee vbox help [COMMAND]                       # Describe subcommands or one specific subcommand
  veewee vbox list                                 # Lists all defined boxes
  veewee vbox ostypes                              # List the available Operating System types
  veewee vbox screenshot [BOX_NAME] [PNGFILENAME]  # Takes a screenshot of the box
  veewee vbox sendkeys [BOX_NAME] [SEQUENCE]       # Sends the key sequence (comma separated) to the box. E.g for testing the :boot_cmd_sequence
  veewee vbox ssh [BOX_NAME] [COMMAND]             # SSH to box
  veewee vbox templates                            # List the currently available templates
  veewee vbox undefine [BOX_NAME]                  # Removes the definition of a basebox 
  veewee vbox up [BOX_NAME]                        # Starts a Box
  veewee vbox validate [BOX_NAME]                  # Validates a box against vagrant compliancy rules
  veewee vbox winrm [BOX_NAME] [COMMAND]           # Execute command via winrm
Options:
          [--debug]           # enable debugging
  -w, --workdir, [--cwd=CWD]  # Change the working directory. (The folder containing the definitions folder).
                              # Default: /tmp/veewee/veewee

Veewee Basics

Template Naming Scheme

Each template folder name follows a naming scheme to help you choosing the right template

<OS name>-<version>-<architecture>[-<install flavor>]

For example, the template for a Ubuntu 12.04 server (amd64) basebox looks like this

ubuntu-12.04-server-amd64[-netboot]
^       ^           ^     ^ 
|       |           |     +----- install flavor                        
|       |           +----- architecture
|       +----- version
+----- OS name

Typical Usage

A common workflow to build a new base box with Veewee is:

• List all templates to find a single template to use
• Define a new box definition from a selected template
• Customize the definition (optional)
• Build a VM using standard ISOs, your own definition settings, and some postinstall scripts
• Validate the new VM
• Manually alter the VM by logging in with ssh (optional; but not recommended)
• Export the VM for distribution or to be used in Vagrant

Getting Help

If you'd like some help with a particular command

$ bundle exec veewee help <command>

And if you're not sure how to use a subcommand, get help with

$ bundle exec veewee <provider> help <subcommand>

The subcommand help is often useful since it will list available optional flag arguments.

List Existing Definitions

To list all available definitions that you've previously created or copied

$ bundle exec veewee <provider> list

Create A Definition

A definition is created by cloning a template into the definitions/ folder.

To create a definition, use the define subcommand

$ bundle exec veewee <provider> define 'precise64' 'ubuntu-12.04.2-server-amd64'

If you want to use an external repository for the definition, you can specify a git URL

$ bundle exec veewee <provider> define 'precise64' 'git://github.com/terrywang/precise64'

Modify a definition (optional)

You can tweak and customize every detail of the box by modifying and extending the (sane) default settings that come with a template. If you want to modify these settings take a look at the Customization instructions.

Remove a definition

If you change your mind and want to get rid of a definition simply call this subcommand

$ bundle exec veewee <provider> undefine 'precise64'

Or you can remove the folder under definitions

$ rm -r ./definitions/precise64

Manage ISO files

The distro ISOs (also called disk images) provide all files needed to install the OS. This file is essential for starting the installation process.

If you already have an .iso file for the desired distribution on your disk, put it inside the iso/ directory and make sure definition.rb is referencing the correct file.

 If an expected ISO is not found in the iso/ directory, Veewee will ask you to download the ISO file from the web. Depending on your internet connection fetching an ISO file can take a while.

Build a new VM image

In order to build the defined box, execute this subcommand

$ bundle exec veewee <provider> build 'precise64'

The build subcommand can take the following optional flags

The build subcommand will run the following routines behind the scenes

• Create a machine and disk according to the definition.rb Note: :os_type_id is the internal name Virtualbox uses for a given distribution
• Mount the ISO file :iso_file
• Boot up the machine and wait for :boot_time
• Send the keystrokes in :boot_cmd_sequence
• Start up a webserver on :kickstart_port to wait for a request from the :kickstart_file IMPORTANT: Do NOT navigate to the file in your browser or the server will stop and the installer will not be able to find your preseed
• Wait for ssh login to work with :ssh_user and :ssh_password
• sudo execute the :postinstall_files

Validate a build

After an OS has been installed on your VM image, you can verify that the machine is configured as intended with the validate subcommand. Veewee provides several tests to help you with that. The tests are located under the validation/ directory.

This subcommand executes all tests on a given machine

$ bundle exec veewee <provider> validate 'precise64'

Validate will run some cucumber tests against the box to see if it has the necessary bits and pieces (e.g. for vagrant to work).

Export a build for distribution

The following subcommand take care of exporting

$ bundle exec veewee <provider> export 'precise64'

The exported filetype depends on the provider. For more details on the providers, please have a look at the Providers doc.

Learn by Example

Make a Oracle Linux 6.4 x86_64 base box compatible with VirtualBox.

Go find the OracleLinux-6.4-x86_64-DVD template

$ bundle exec veewee vbox templates | grep -i oracle
veewee vbox define '<box_name>' 'OracleLinux-5.9-x86_64-DVD' --workdir=/data/DevOps/veewee/veewee
veewee vbox define '<box_name>' 'OracleLinux-6.3-x86_64-DVD' --workdir=/data/DevOps/veewee/veewee
veewee vbox define '<box_name>' 'OracleLinux-6.4-x86_64-DVD' --workdir=/data/DevOps/veewee/veewee

Then use the define command to create a new definition with a custom name. The following command copies the folder templates/OracleLinux-6.4-x86_64 to definitions/oracle-6.4-x86_64

$ bundle exec veewee vbox define 'oracle-6.4-x86_64' 'OracleLinux-6.4-x86_64-DVD'
The basebox 'oracle-6.4-x86_64' has been successfully created from the template 'OracleLinux-6.4-x86_64-DVD'
You can now edit the definition files stored in /data/DevOps/veewee/veewee/definitions/oracle-6.4-x86_64 or build the box with:
veewee vbox build 'oracle-6.4-x86_64' --workdir=/data/DevOps/veewee/veewee

IMPORTANT: You should avoid dots in the name because the box name gets used as the hostname also. Dots in the box name currently lead to invalid hostnames which causes several negative side effects (e.g. preventing the network devices to start).

Confirm that all expected files are in place

$ ls definitions/oracle-6.4-x86_64
base.sh  chef.sh  cleanup.sh  definition.rb  ks.cfg  proxy.sh  puppet.sh  ruby.sh  vagrant.sh  virtualbox.sh  zerodisk.sh

You can now inspect and optionally customize the defaults.

Next it's time to build the VM image with

$ bundle exec veewee vbox build 'oracle-6.4-x86_64'
# Enable debug mode output
$ bundle exec veewee vbox build 'oracle-6.4-x86_64' --debug

Veewee now asks to download the distro ISO (unless --auto is provided) and will start creating the VM image. This process can take some time.

After the build completes, you can run the provided test suite on your new VM

$ bundle exec veewee vbox validate 'oracle-6.4-x86_64'

Validation is highly recommended before requesting a fork pull on any modified templates.

Finally let's export the box so it can be distributed or used by Vagrant

$ bundle exec veewee vbox export 'oracle-6.4-x86_64'

Customizing Definitions

Customize

Definition overview

Definitions are stored under the definitions/ directory relative to the current directory.

.
├── definitions
│   └── myubuntubox
│       ├── <preseed.cfg, kickstart.cfg, ...>
│       ├── base.sh
│       ├── cleanup.sh
│       ├── chef.sh
│       ├── puppet.sh
│       ├── ruby.sh
│       ├── virtualbox.sh
│       └── ....sh
└── README.md 

The file definition.rb contains all the parameters to define the machine to be build (see below)

• memory size
• number of CPUs
• user account and password
• sudo command
• shutdown command
• URL and checksum to download the ISO

When a new machine boots, it will typically fetch its initial configuration file over http from a kickstart file defined in kickstart_file. These files are usually named preseed.cfg or ks.cfg.

Postinstall scripts

You can define multiple postinstall files by providing an array of filenames within definition.rb, like so

:postinstall_files => [ "postinstall.sh", "postinstall_2.sh" ],

Once the initial installation is done, Veewee will execute each postinstall .sh file on the machine in chronologic order (order found in :postinstall_files array).

The main reason for splitting up the original postinstall.sh script into multiple files is to make the post-install steps as reusable and portable as possible for different virtualization systems and/or operating systems. For example, there is no need to install the Virtualbox Guest Additions on KVM or VMware Fusion.

Postinstall barebones

A definition usually consists of at least these postinstall files

Newer definitions contain of even more files (they have broken postinstall.sh into multiple files) to get a finer separation of concerns for the installation.

Using ERB in Files

Add .erb to your files in a definition and they will get parsed accordingly.

This is useful for generating kickstart, post-install at runtime.

Configuring definition.rb

The definition.rb file is the core definition file of each box. All crucial properties and postinstall scripts are defined here.

The boot_cmd_sequence parameter allows you to override the initial commands (like keyboard layout) that are fired up in the first boot sequence.

All other settings are used internally by Veewee, the virtualization provider, or simply for choosing the proper ISO

Veewee::Definition.declare( {
    :cpu_count => '1',
    :memory_size=> '256',
    :disk_size => '10140',
    :disk_format => 'VDI',
    :disk_variant => 'Standard',
    :os_type_id => 'Ubuntu',
    :iso_file => "ubuntu-12.10-server-i386.iso",
    :iso_src => "http://releases.ubuntu.com/precise/ubuntu-12.10-server-i386.iso",
    :iso_md5 => "3daaa312833a7da1e85e2a02787e4b66",
    :iso_download_timeout => "1000",
    :boot_wait => "10",
    :boot_cmd_sequence => [
        '<Esc><Esc><Enter>',
        '/install/vmlinuz noapic preseed/url=http://%IP%:%PORT%/preseed.cfg ',
        'debian-installer=en_US auto locale=en_US kbd-chooser/method=us ',
        'hostname=%NAME% ',
        'fb=false debconf/frontend=noninteractive ',
        'console-setup/ask_detect=false console-setup/modelcode=pc105 console-setup/layoutcode=us ',
        'initrd=/install/initrd.gz -- <Enter>'
    ],
    :kickstart_port => "7122",
    :kickstart_timeout => "10000",
    :kickstart_file => "preseed.cfg",
    :ssh_login_timeout => "10000",
    :ssh_user => "vagrant",
    :ssh_password => "vagrant",
    :ssh_key => "",
    :ssh_host_port => "2222", :ssh_guest_port => "22",
    :sudo_cmd => "echo '%p'|sudo -S sh '%f'",
    :shutdown_cmd => "shutdown -H",
    :postinstall_files => [ "postinstall.sh"],
    :postinstall_timeout => "10000"
})

IMPORTANT: If you change values directly in a template, be sure to run bundle exec veewee <provider> undefine to remove the old definition and then bundle exec veewee <provider> define again to copy the updated template files into the definition.

If you are an experienced devops veteran and have enhanced template settings, please let us know why. We are very interested in improving Veewee's templates.

Provider vm_options

Each provider can take options that are specific the provider; more details will be available in each provider doc but let's have a quick overview here

Veewee::Definition.declare({
    :cpu_count => '1',
    :memory_size => '256',
    :disk_size => '10140',
    :disk_format => 'VDI',
    :disk_variant => 'Standard',
    # […]
    :postinstall_files => [ "postinstall.sh" ],
    :postinstall_timeout => "10000",
    :kvm => {
        :vm_options => [
            'network_type' => 'bridge',
            'network_bridge_name' => 'brlxc0'
        ]
    },
    :virtualbox => {
        :vm_options => [
            'pae' => 'on',
            'ioapic' => 'one'
        ]
    }
})

This box will have pae and ioapic enabled with VirtualBox, and will use the brlxc0 bridge with KVM (on libvirt).

Veewee Fundamentals

Veewee Basics

• Veewee Basics covers creating standard-issue boxes
  https://github.com/jedi4ever/veewee/blob/master/doc/basics.md 
• Customizing Definitions helps you fine tune each box definition to meet your exact needs
  https://github.com/jedi4ever/veewee/blob/master/doc/customize.md
  
• Veeweefile can be used to define your own paths
  https://github.com/jedi4ever/veewee/blob/master/doc/veeweefile.md
  

Guide for Vagrant

Guide for VirtualBox

Veewee Definitions Examples

Opscode bento and Puppet Labs Vagrant Boxes

•  Opscode bento
  https://github.com/opscode/bento
• Puppet Labs Vagrant Boxes
  https://github.com/puppetlabs/puppet-vagrant-boxes

Reference

https://github.com/jedi4ever/veewee