How to use virt-install in OpenSolaris

Booting up dom0

To use xVM, install OpenSolaris 76 or later and boot the system as dom0. After you
install 76, you can control the mode the system runs in by chosing from the
grub menu at boot-up or by setting the default boot mode in menu.lst.
Menu.lst has entries for each mode. In build 76, they are:

title Solaris Express Community Edition snv_76 X86 [0]
title Solaris xVM [1]
title Solaris failsafe [2]
title Diagnostic Partition [3]

At the top of the file (/boot/grub/menu.lst) you'll find the default boot
definition:

# default menu entry
default 0

You can change the entry from the initial default (regular Solaris on metal) to
xVM (dom0) by changing the default menu entry to 1:

# default menu entry
default 1

After rebooting, you should be able to run xm commands to check the status of
the xVM system:

# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 727 2 r----- 624.7

In this case, there are no domUs (guests) running.

PV vs. HVM

There are two kinds of guests that can run on dom0: paravitualized and fully
virtualized (Hardware Virtual Machine). Paravirtualized guests have OS
support which allow them to run. HVM guests run without change and can only
do so on systems which have appropriate hardware.

Installing Paravitualized DomUs

Virt-install is the recommended way to create guests. Run virt-install as root
on dom0.

The simplest domU install is a paravitual install from an .iso file to a file
or a physical partition. The example here is to a file (so you can see the
mechanics of how to do it). It's a lot faster to use a partition, however, so
they are the preferred Virtual Backing Device (VBD). More on that in a later
example.

virt-install -n domu-x17 /
--nographics -p /
-f /xen2/nv76.img -s 20 /
-l /net/netinstall-sca11/export/XenDomU/x_iso/nv_76/nv_76.iso /
-r 1280

-n name or handle of the domU. This is usually both the handle used by the
dom0 to reference the domU and its hostname. This is not always the case
though, as explained below.

-p --nographics This is a paravirtual install [-p]. The console is serial.
These two options are always together. You may not do a paravirtual
install with a graphics console.

-f The Virtual Backing Device (VBD). In this case, it's a file. The file
is created by virt-install unless it already exists. By default, virt-
install creates a sparse image file. You can have it create a non-sparse
image by using --nonsparse on the virt-install command line. The VBD can
be on a remote system (if you have permissions) and is specified with the
/net/system construction.

A physical partition or device can be used, such as /dev/dsk/c0t0d0s3.

-s The size of the VBD in GB. In this case, the partition is 20 GB.

-l The OS Image. This can also be a physical device (DVD) or the path to an
ISO file. If you are using an ISO on a remote server, you must use the
/net/blah form. For instance:

/net/netinstall-sca11/export/nv/x_iso/76/nv_76.iso

If you are specifying an install image directory the format has to be:

nfs:server:path

The server name has to be for the interface card over which installs occur,
not just the server's name. For example netinstall-sca11-195 (net 195)
NOT netinstall-sca11. You can also use the IP address for the NIC.

-r Memory to use for the domU, in MB. Use 1 GB or more. Go under 1 GB and
you may see hangs.

This is a simple virt-install to a filed-backed VBD.

When you do a non-jumpstart virt-install, an Solaris Interactive Text Console
Session is started on your behalf.

Go through the interactive session, specifying the usual configuration values.
Note that through the interactive session, you can specify data that is not
consistent with your virt-install command.

Here's a synopsis of the interesting values

Network Connectivity: Yes
DHCP: No, but you can specify yes. If you do and your name server is setup to
supply DHCP, your domU will assigned a host name. Note that this will
obviously NOT be the same as the name you used with -n in virt-install.
When you refer to this domU with xm or any xVM utility, you must use the
value you specified with -n. Once your domU is up and running, you can
ssh or rlogin to it using the name assigned by dhcp.
Host Name: domu-x17 (This can be anything you want as long as you specify the
IP address for this system in the next question.
Host IP Address: 10.4.195.218
Name Service: NIS
Domain Name: scalab.sfbay.sun.com
NFSv4 Domain: sun.com
Solaris Interactive Install: Standard, Flash, Upgrade all work
Standard is your usual Standard experience, as is flash. Note that like
any other flash install, your flash archive should match the version of
install being run to do the installation. In other words, the source you
specify with -l in virt-install should match your flash archive.

Upgrade is tricky. If you try to virt-install over an existing domU,
virt-install won't let you unless you trick it into thinking initially
that a domU doesn't already exist. To do this, you have to run:

# virsh undefine domUname

Then do your virt-install. Select Upgrade when you get to this screen.
You can also use jumpstart to upgrade by using a profile that does an
upgrade instead of an initial install. You also have to undefine the
domU if you do a jumpstart upgrade.
Choose Media: CD/DVD
Select Software: Choose whichever metacluster you want
Select Disks: Choose the default
Create Solaris fdisk Partition: Yes
This disk will be the one you define with virt-install options -f and -s
Disk Layout: Chose auto or manual like any other install
Then go ahead and have it install like any other system.

While the domU installs, you can see its status:

# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 725 2 r----- 172.3
domu-x17 1 1279 1 -b---- 236.5

When your guest completes installing, the system will reboot (well, it'll reboot
if you've specified auto-reboot to suninstall):

rebooting...

Guest installation complete... restarting guest.
SunOS Release 5.11 Version snv_76 64-bit
Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Hostname: domu-x17
Configuring devices.
Loading smf(5) service descriptions: 179/179
NIS domain name is scalab.sfbay.sun.com
Reading ZFS config: done.
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair

domu-x17 console login:

From dom0, it looks like this:

# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 727 2 r----- 324.3
domu-x17 2 1279 1 -b---- 393.1

Using Jumpstart with Virt-Install to Create PV DomUs

Jumpstarting PV domUs is fundamentally the same as jumpstarting Solaris on
hardware. At present, there is only specific support to jumpstart PV guests in
virt-install. You can use custom jumpstart from Solaris Install when creating
HVM guests; however, this is part of the install subsystem and not virt-install.

This example uses a disk partition to store the guest image. The fastest
installs of domUs are to devices as opposed to files. Installs to files use
lofs, which is slow. Installing from ISOs directly is also slow so we generally
do netinstalls or jumpstarts to physical partitions.

Note: when installing to a partition, install sometimes tries to interpret the
existing data, so zero out the first block of data:

#> dd if=/dev/zero of=/dev/dsk/c0t0d0s3 count=2000

Also, disable any entry in /etc/vfstab for the partition you're using to prevent
the dom0 from trying to mount it when it's rebooted.

As with jumpstart installs of Solaris to hardware, you have to configure your
install server for the guests you jumpstart with virt-install. This involves:

o Setting up rules files and profiles in an exported directory
o Getting your image set up
o Doing and addclient or add_install_client

Setting up and install server is documented in the Solaris manuals and won't be
covered here.

On your install server, you need to do an add_install_client to enable the
jumpstart of your guest, in this case domu-x12:

# uname -a
SunOS netinstall-sca11 5.11 snv_69 i86pc i386 i86pc

As root, change your directory to the Tools area of the netinstall image to
be installed on the guest:

# cd /export/nv/x/76/Solaris_11/Tools

# Run add_install_client for the guest, domu-x12
# add_install_client -e aa:ff:bb:aa:12:aa /
-b console=ttya /
-s 10.4.195.10:/export/nv/x/76 /
-t /export/nv/x/76/boot /
-c netinstall-sca11-195:/export/wuwu/js/x/domu-x12 /
-p netinstall-sca11-195:/export/wuwu/js/x/domu-x12/foo1 /
domu-x12 i86pc

Ethernet Address (-e):
Should be the one specified in your (subsequent) virt-install command line
with -m. These must match. Check /etc/ethers to make sure the address is
either not present or correct for domU to be installed. Ethernet addresses
for domUs are made up and have to be unique. If you do a virt-install that
isn't a jumpstart, you don't have to specify a mac address -- a unique one
will be assigned automatically. When you do a jumpstart, however, you must
specify one because you've got to specify the same address to virt-install
that you use when you do the addclient.

Server Path (-s):
NFS-style path to the install image location. The IP address used should
correspond to the NIC used on the install server to service the network
on which the dom0 and domU reside. This corresponds to the -l option that
will be specified in the virt-install command used to create domu-x12.

netinstall-sca11:root> ls /export/nv/x/76
Copyright Solaris_11/
JDS-THIRDPARTYLICENSEREADME boot/
License/ installer*

If the -s argument in your addclient does not match the -l argument in your
virt-install, then for now, -l wins. This behavior is not guaranteed so
it's best if -l and -s point to the same bits.

Install Boot Image Patch (-t):
The path to the boot directory of the install image.

If the -t argument in your addclient does not match the -l argument in your
virt-install, then for now, -l wins. This behavior is not guaranteed so
it's best if -l and -s point to the same bits.

Jumpstart Directory (-c):

NFS path to the jumpstart directory containing the rules.ok file specifying
the location of the jumpstart profile and finish script. This should be
the same value you pass to virt-install with the --autocf option.

netinstall-sca11:root> ls -l /export/wuwu/js/x/domu-x12
total 82
-rwsr-xr-x 1 root root 59701 Sep 10 10:32 check*
-rwxr-xr-x 1 root root 336 Sep 10 10:33 domu.profile*
-rwxr-xr-x 1 root root 644 Sep 12 14:18 finish*
-rw-r--r-- 1 root root 40 Sep 10 10:32 rules
-rw-r--r-- 1 root root 66 Sep 28 15:33 rules.ok
drwxr-xr-x 2 root root 3 Sep 28 15:33 sysid/

netinstall-sca11:root> cat /export/wuwu/js/x/domu-x12/domu.profile
install_type initial_install
system_type standalone
cluster SUNWCXall

fdisk rootdisk solaris maxfree

partitioning explicit
filesys rootdisk.s0 free /
filesys rootdisk.s1 2560 swap

Note: The profile should layout the filesystems consistent with the
size given with the -s option in virt-install. If, for instance,
-s 30 is used, the total space allocated in the profile should be
under 30 GB.

Note: Both -p and -c values override values specified in the virt-install
command. They are included in this documentation because
there are ways of doing custom jumpstarts in which virt-install isn't
used. In these cases, -p and -c need to be specified in order for the
sysidcfg and jumpstart directories to be found.

Directory containing the sysidcfg file (-p):

NFS path to the directory containing the sysidcfg file to use.

netinstall-sca11:root> ls -l /export/wuwu/js/x/domu-x12/sysid
total 2
-rw-r--r-- 1 root root 307 Sep 10 10:32 sysidcfg

The add_install_client will create an entry in /etc/bootparams containing all
the info needed to effect the jumpstart from the server side. Here's the entry
created by the add_install_client call used in this example:

domu-x12 root=netinstall-sca11-195: install=10.4.195.10:/export/nv/x/76
boottype=:in sysid_config=netinstall-sca11:/export/wuwu/js/x/domu-x12/sysid in
stall_config=netinstall-sca11:/export/wuwu/js/x/domu-x12 rootopts=:rsize=8192

It also adds entries to /etc/hosts and /etc/ethers.

If something goes wrong with your jumpstart, you can look at /etc/bootparams to
make sure it appears as expected and that there aren't duplicate entries. Ditto
for /etc/hosts and /etc/ethers.

Okay, now to do the virt-install:

sol-x2100-12:root> virt-install -n domu-x12 /
--nographics -p /
-f /dev/dsk/c0t0d0s3 -s 20 /
-l nfs:netinstall-sca11-195:/export/nv/x/76 /
-m aa:ff:bb:aa:12:aa /
-r 1280 /
--autocf "nfs:netinstall-sca11-195:/export/wuwu/js/x/domu-x12"

Notes:
-p --nographics This is a paravirtual install [-p]. The console is serial.
These two options are always together. You may not do a paravirtual
install with a graphics console.

-n name or handle of the domU.

-f The Virtual Backing Device (VBD). In this case a partition.

-s The size of the VBD in GB. In this case, the partition is 20 GB.

-l The netinstall image location. For netinstalls, the format has to be
nfs:server:path. The server name has to be for the interface card over
which install occurs, not just the server's name. In the example, it's
netinstall-sca11-195 (net 195) NOT netinstall-sca11. You can also use the
ethernet address, for example 10.4.195.10.

-m Ethernet ID. The ethernet address you used in the addclient to configure
the server side.

-r Memory to use for the domU, in MB. Use 1 GB or more. Go under 1 GB and
you may see hangs.

--autocf spec
The jumpstart directory. Virt-install does not let you specify a separate
directory for the sysidcfg and the jumpstart directory, but you can do that
with the -p and -c options to addclient. Note in the output that although
the virt-install autocf option specifies domu-x12 as the sysidcfg directory
the sysidcfg file is taken from the domu-x12/foo1 directory, which was
specified in the addclient command.

Output:
sol-x2100-12:root> virt-install -n domu-x12 /
> --nographics -p /
> -f /dev/dsk/c0t0d0s3 -s 20 /
> -l /net/netinstall-sca11/export/XenDomU/x_iso/nv_76/nv_76.iso /
> -m aa:ff:bb:aa:12:aa /
> -r 1280 /
> --autocf "nfs:netinstall-sca11-195:/export/wuwu/js/x/domu-x12"


Starting install...
Creating domain... 0 B 00:09
SunOS Release 5.11 Version snv_76 64-bit
Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Configuring /dev
Custom JumpStart
Using install cd in /dev/dsk/c0d1p0
Using RPC Bootparams for network configuration information.
Attempting to configure interface xnf0...
Configured interface xnf0
Reading ZFS config: done.
Setting up Java. Please wait...
Beginning system identification...
Searching for configuration file(s)...
Using sysid configuration file 10.4.195.10:/export/wuwu/js/x/domu-x12/foo1/sysidcfg
Search complete.
Discovering additional network configuration...
Completing system identification...
Starting remote procedure call (RPC) services: done.
NIS domain name is scalab.sfbay.sun.com
System identification complete.
Starting Solaris installation program...
Searching for JumpStart directory...
Using rules.ok from 10.4.195.10:/export/wuwu/js/x/domu-x12.
Checking rules.ok file...
Using profile: domu.profile
Using finish script: finish
Executing JumpStart preinstall phase...
Searching for SolStart directory...
Checking rules.ok file...
Using begin script: install_begin
Using finish script: patch_finish
Executing SolStart preinstall phase...
Executing begin script "install_begin"...
Begin script install_begin execution completed.

Processing default locales
- Specifying default locale (en_US.ISO8859-1)

Processing profile
- Selecting cluster (SUNWCXall)
- Selecting locale (en_US.ISO8859-1)
- Selecting all disks
- Configuring boot device
- Using disk (c0d0) for "rootdisk"
- Creating "maxfree" Solaris fdisk partition (c0d0)
- Configuring swap (c0d0s1)
- Configuring / (c0d0s0)

Verifying disk configuration

Verifying space allocation
- Total software size: 4011.42 Mbytes

Etc.
=============================
Installing HVM Guests

HVM installs of Solaris differ from PV guests in the following ways:

o You must use a graphics console, which in the case of this demo, is vnc
o The -l virt-install option isn't used. HVM installs use -c to specify
the image source, which must be an ISO file, not a netinstall directory
o Jumpstart installs cannot be initiated directly through virt-install

Before you initiate virt-install, you must manually set the vnc root password.
This is the password VNC will use to allow access to the console when the
console session starts.

# Set the prop in xvm which is used to initialize the console password
soe-x4600m2-2:root> svccfg -s xvm/xend setprop /
config/vncpasswd=astring:/"bacon/"
# Restart xend
soe-x4600m2-2:root> svcadm refresh xvm/xend; svcadm restart xvm/xend

In the example below, Nevada 76 is being installed. You could also install
S10U4, Windows XP, Linux, or any of the other supported operating systems. For
more on this, please consult the matrix website:

http://perf.eng.sun.com/twiki/bin/view/Matrix

soe-x4600m2-2:root> export DISPLAY=wu:0
soe-x4600m2-2:root> virt-install -d -n domu232-1 /
-c "/net/netinstall-sca11/export/XenDomU/x_iso/nv_76/nv_76.iso" /
-s 34 /
-r 2000 --hvm /
-m aa:aa:bb:cc:dd:1a /
-f /dev/dsk/c3t1d0s3 /
--vnc &

You don't have to run the command in background. I use background mode to keep
the terminal session free.

This brings up a vnc password popup. Enter the password you set above and the
console will come up. If you make a mistake and kill the password box, you can
restart the console with:

soe-x4600m2-2:root> vncviewer 127.0.0.1:0 &

Multiple consoles are created by incrementing the number to the right of the
colon in 127.0.0.1:0. The second console created by vncviewer would be
127.0.0.1:1.

The first screen to come up is the grub menu. Choose the second option in the
menu for now (SXCE), which is the current day sun installer. The top entry
(SXDE) is the dwarf caiman installer, which was having some problems.

Go through the interactive install, just as if you were installing a system.

Eventually, the guest will finish installing and reboot. After the reboot, a
dtlogin session will come up in the VNC viewer.

Keep in mind that installing ISOs takes a LONG time. At a certain point, you'll
wait for maybe twenty or thirty minutes waiting for things to configure. It's
best to just go ahead and do something else. When I did my first install, I
kept aborting it thinking things were hung up.

HVM Jumpstart Installs

While you can't initiate HVM jumpstarts directly through virt-install, you CAN
do a jumpstart. After the console comes up and sun install is running, choose
Option 2, Custom Jumpstart.

First set up the server for the jumpstart:

netinstall-sca11:root> cd /export/nv/x/latest/Solaris_11/Tools
netinstall-sca11:root> add_install_client -e aa:aa:bb:cc:dd:2a /
-b console=ttya /
-s 10.4.195.10:/export/nv/x/76 /
-t /export/nv/x/76/boot /
-c netinstall-sca11:/export/wuwu/js/x/domu232-2 /
-p 10.5.232.10:/export/wuwu/js/x/domu232-2 /
domu232-2 i86pc

Then do your virt-install:

netinstall-sca11:root> virt-install -d -n domu232-2 /
-c "/net/netinstall-sca11/export/XenDomU/x_iso/nv_76/nv_76.iso" /
-s 34 /
-r 2000 --hvm /
-m aa:aa:bb:cc:dd:2a /
-f /dev/dsk/c3t1d0s4 /
--vnc &

Choose custom jumpstart from menu. The install should then proceed without
the need for any intervention on your part.

Using custom jumpstart is at least twice as fast as installing from ISOs. You
can save yourself a lot of grief by using it.

Cleaning Up Botched Installs

When virt-installs abort, you will sometimes find domUs left in a incomplete
state. The situation is analagous to what happens when a regular install blows
up. Suninstall exits and you are left at the root prompt. To destroy the
domU to try again use the following sequence from dom0:

# xm shutdown domUname
# xm destroy domUname
# virsh undefine domUname

Jumpstart Installs

If you are having problems with jumpstart installs, check the following:

o Make sure you specified the correct mac addresses to both virt-install
and add_install_client

o NICs you specify with nfs: construction must be on the same subnet as
your domU, which must be on the same subnet as your dom0

o Make sure there are no conflicting entries in /etc/bootparams,
/etc/hosts or /etc/ethers

o You have set up your jumpstart directory correctly and your rules file
has a rule for your domU

o Your sysidcfg file contains the correct default route for the network
on which your domU resides

o Your install server doesn't have any network-related services down

o DomUs IP addresses have been pushed to the NIS servers