Solaris: UFS to ZFS, LiveUpgrade and Patching
This article gives a detailed overview, how we migrate our servers from UFS to ZFS boot 2-way mirros, how they are upgraded to Solaris™ 10u6 aka 10/08 with /var on a separate ZFS and finally how to accomplish "day-to-day" patching. The main stages are devided into:
- Moving from UFS to ZFS boot
- Cleaning up the old UFS Boot Environment
- Upgrade to S10u6
- Cleanup - Phase 2 (ZFS)
- day-to-day patching (ZFS)
Make sure, that on each stage all zones are running or at least bootable, and the environment variables show below are properly set. Also give your brain a chance and think before you blindly copy-and-paste any commands mentioned in this article! There is no guarantee, that the commands shown in this article match exactly your system and thus may damage it/cause data loss, if you do not adjust them to your needs!
'til now, the shown procedures have been successfully tested on a Sun Fire V240 server (2 GB RAM, 2x1.5GHz UltraSPARC-IIIi, 2x36GB HDDi, 2x 72GB HDDi) with one running sparse zone, only. However, our plan is to migrate ~93% of our servers (Sun Fire 280R, 420R, V440, V490, T1000, X4500, X4600) to S10u6 an thus any further problems we see, will be reflected in this article - i.e. watch out for changes!
setenv CD /net/install/pool1/install/sparc/Solaris_10_u6-ga1
setenv JUMPDIR /net/install/pool1/install/jumpstart
mount /local/misc
set path = ( /usr/bin /usr/sbin /local/misc/sbin )
Moving from UFS to ZFS boot
update to S10u6 aka 10/08 via recommended/feature patching
on pre U4 systems SUNWlucfg is probably missing:
pkgadd -d $CD/Solaris_10/Product SUNWlucfg
make sure, that the following patches are installed:
137137-09 - U6 kernel Patch and its lu/zfs boot dependencies:
119252-26, 119254-59, 119313-23, 121430-29, 121428-11, 124628-08, 124630-19
see also: Solaris™ Live Upgrade Software: Minimum Patch Requirements
see also: checkpatches.sh
see also: Creating a Boot Environment for ZFS Root Pools
Solaris 10 10/08 Release and Installation Collection >> Solaris 10 10/08 Release Notes >> 2. Solaris Runtime Issues >> File Systems
determine the HDD for the new root pool aka rpool
echo | format
In this example we use: c0t1d0
format the disk so that the whole disk can be used by ZFS
# on x86
fdisk -B /dev/rdsk/c0t1d0p0
# on sparc delete all slices and assign all blocks to s0
format -d c0t1d0If you want to use mirroring, make sure, that s0 of HDD0 and HDD1 have finally the same size (use number of blocks when specifying its size)
Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 29772 34.41GB (29773/0/0) 72169752
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 29772 34.41GB (29773/0/0) 72169752
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0move all UFS zones to ZFS mirror 'pool1' on HDD2 and HDD3
This allows LU to use zone snapshots instead of copying the stuff - thus magnitudes faster. In our example, UFS zones are in /export/scratch/zones/ ; pool1 mountpoint is on /pool1 .
One may use the following ksh snipplet (requires GNU sed!):
ksh
zfs create pool1/zones
# adjust this and the following variable
UFSZONES="zone1 zone2 ..."
UFSZPATH="/export/scratch/zones"
for ZNAME in $UFSZONES ; do
zlogin $ZNAME 'init 5'
done
echo 'verify
commit
' >/tmp/zone.cmd
for ZNAME in $UFSZONES ; do
# and wait, 'til $ZNAME is down
while true; do
zoneadm list | /usr/xpg4/bin/grep -q "^$ZNAME"'$'
[ $? -ne 0 ] && break
done
zfs create pool1/zones/$ZNAME
mv $UFSZPATH/$ZNAME/* /pool1/zones/$ZNAME/
chmod 700 /pool1/zones/$ZNAME
gsed -i /
-e "/zonepath=/ s,$UFSZPATH/$ZNAME,/pool1/zones/$ZNAME," /
/etc/zones/$ZNAME.xml
zonecfg -z $ZNAME -f /tmp/zone.cmd
zoneadm -z $ZNAME boot
done
exitcreate the rpool
zpool create -f -o failmode=continue rpool c0t1d0s0
# some bugs? require us to do the following manually
zfs set mountpoint=/rpool rpool
zfs create -o mountpoint=legacy rpool/ROOT
zfs create -o canmount=noauto rpool/ROOT/zfs1008BE
zfs create rpool/ROOT/zfs1008BE/var
zpool set bootfs=rpool/ROOT/zfs1008BE rpool
zfs set mountpoint=/ rpool/ROOT/zfs1008BEcreate the ZFS based Boot Environment (BE)
lucreate -c ufs1008BE -n zfs1008BE -p rpool
~25min on V240
At least here one probably ask itself, why we do not use pool1 for boot, then form a mirror of HDD0 and HDD1 and put another BE on the mirror. The answer is pretty simple: because some machines like the thumper aka X4500 can boot from 2 special disks, only (c5t0d0 and c5t4d0).
move BE's /var to a separate ZFS within the BE
zfs set mountpoint=/mnt rpool/ROOT/zfs1008BE
zfs mount rpool/ROOT/zfs1008BE
zfs create rpool/ROOT/zfs1008BE/mnt
cd /mnt/var
find . -depth -print | cpio -puvmdP@ /mnt/mnt/
rm -rf /mnt/mnt/lost+found
cd /mnt; rm -rf /mnt/var
zfs rename rpool/ROOT/zfs1008BE/mnt rpool/ROOT/zfs1008BE/var
zfs umount rpool/ROOT/zfs1008BE
zfs set mountpoint=/ rpool/ROOT/zfs1008BE
zfs set canmount=noauto rpool/ROOT/zfs1008BE/var~7 min on V240
activate the new ZFS based BE
luactivate zfs1008BE
copy the output of the command to a safe place, e.g. USB stick
restart the machine
init 6
after reboot, check that everything is ok
E.g.:
df -h
dmesg
# PATH should be /pool1/zones/$zname-zfs1008BE for none-global zones
zoneadm list -iv
lustatus
Cleaning up
destroy old UFS BE
ludelete ufs1008BE
One will get warnings about not beeing able to delete ZFSs for the old bootenv like /.alt.tmp.b-LN.mnt/pool1/zones/$zname - that's ok. Furthermore one can promote its clones like /pool1/zones/$zname-zfs1008BE and than remove the old ones including their snapshots on desire.
make sure, everything is still ok
init 6
move all remaining filesystems from HDD0 to the root pool
Depending on the mount hierarchy, the following recipe needs to be adapted!
a) check
df -h | grep c0t0d0
b) stop all zones and processes, which use those UFS slices (remember to
unshare those slices, if exported via NFS)
zlogin $ZNAME 'init 5'
c) create appropriate ZFSs
foreach USLICE in $UFS_SLICES_FROM_a
zfs create rpool/mnt
# just to be sure
mount -o ro -o remount $USLICE.mntpoint
cd $USLICE.mntpoint
find . -depth -print | cpio -puvmdP@ /rpool/mnt/
rm -rf /rpool/mnt/lost+found
umount $USLICE.mntpoint
# comment out the appropriate entry in /etc/vfstab
gsed -i -e "/^$USLICE/ s,^,#," /etc/vfstab
zfs rename -p rpool/mnt rpool/$USLICE.mntpoint
# in case of NFS export, comment out entries in /etc/dfs/dfstab
# and apply to the ZFS itself
zfs set sharenfs='rw=sol:bsd:lnx,root=admhosts' rpool/mnt
# if the parent mountpoint is not appropriate
zfs set mountpoint=$USLICE.mntpoint rpool/$USLICE.mntpoint
doneadjust /etc/lu/ICF.$NUM
Deduce $NUM from /etc/lutab (e.g. with grep :`lucurr`: /etc/lutab | cut -f1 -d:), replace c0t0d0* entries with its zfs counter part and add all parents not yet part of that file. E.g. our diff would be:
zfs1008BE:-:/dev/zvol/dsk/rpool/swap:swap:4196352
zfs1008BE:/:rpool/ROOT/zfs1008BE:zfs:0
zfs1008BE:/var:rpool/ROOT/zfs1008BE/var:zfs:0
zfs1008BE:/pool1:pool1:zfs:0
zfs1008BE:/rpool:rpool:zfs:0
zfs1008BE:/pool1/zones:pool1/zones:zfs:0
-zfs1008BE:/var/log/web:/dev/dsk/c0t0d0s6:ufs:16780016
+zfs1008BE:/rpool/var:rpool/var:zfs:0
+zfs1008BE:/rpool/var/log:rpool/var/log:zfs:0
+zfs1008BE:/var/log/web:rpool/var/log/web:zfs:0
-zfs1008BE:/export/scratch:/dev/dsk/c0t0d0s7:ufs:16703368
+zfs1008BE:/export:rpool/export:zfs:0
+zfs1008BE:/export/scratch:rpool/export/scratch:zfs:0make sure, that no slice of HDD0 is used anymore and everything works as expected
init 6
# after reboot
df -h | grep c0t0d0
zpool status
swap -lDestroy any zpool/zfs/volume, which is still assigned to c0t0d0 - if one is still in use, zfs/zpool will warn you.
repartition HDD0 as one slice as described above
attach HDD1 to HDD0 - form a ZFS 2-way mirror
zpool attach rpool c0t1d0s0 c0t0d0s0
Finally, to avoid an unbootable environment, check the ZFS Troubleshooting Guide to fix any known ZFS Boot Issues immediately.
Live Upgrade to S10 U6
See also:
- Solaris Live Upgrade How-To Guide
- Solaris™ Live Upgrade Software: Minimum Patch Requirements
remove the old Live Upgrade packages
pkgrm SUNWluu SUNWlur SUNWlucfg
add the Live Upgrade packages from ufs2zfsthe release to install
pkgadd -d $CD/Solaris_10/Product SUNWluu SUNWlur SUNWlucfg
check, whether all required patches are installed
E.g. wrt. Solaris™ Live Upgrade Software: Minimum Patch Requirements:
checkpatches.sh 119081-25 124628-05 ...
create the new root pool on HDD0
This is usually not neccessary, if you already have a ZFS mirrored boot environment (in this case just use rpool instead of rpool0 in the following examples/scripts and omit this step). However, if e.g. s0 of HDD1 is smaller than s0 of HDD0, the latter one can not be attached to the former one. So we need to "swap" the situation it.
zpool create -o failmode=continue rpool0 c0t0d0s0
zpool statuscheck and fix basic ownership/permissions
pkgchk -v SUNWscpr
create a new boot environment for upgrade
rmdir /.alt.lucopy.*
lucreate -n s10u6 -p rpool0~30 min on V240
mount the new bootenv on /mnt - fix any errors
Do not continue before it executes without any errors
lumount -n s10u6 /mnt
determine patches, which would be removed by luupgrade
can be used to re-apply them after upgrade, if necessary
$CD/Solaris_10/Misc/analyze_patches -N $CD -R /mnt /
>/mnt/var/tmp/s10u6-rm.txt
luumount s10u6~4 min on V240
create the profile to use for the upgrade
We use:
$JUMPDIR/mkProfile.sh $CD/Solaris_10
# remove U6 zone poison SUNWdrr on sparc
echo 'cluster SUNWC4u1 delete' >> /tmp/profile.orig
echo 'cluster SUNWCcvc delete' >> /tmp/profile.orig
# remove/adjust disallowed instructions (there are more, but not in ourxi
# case ;-)) - see luupgrade(1M)
/usr/bin/egrep -v '^(system_type|pool|bootenv|filesys)|nfs:' /
/tmp/profile.orig | /
sed -e '/^install_type/ s,initial_install,upgrade,' /
>/var/tmp/profile
# just if one wanna know the intial setup of the system
cp /var/tmp/profile /mnt/var/sadm/system/logs/profile.s10u6verify
rm -f /var/tmp/upgrade.err /var/tmp/upgrade.out
luupgrade -N -u -n s10u6 /
-l /var/tmp/upgrade.err -o /var/tmp/upgrade.out /
-s $CD -j /var/tmp/profile -Dupgrade if /var/tmp/upgrade.err is empty
rm -f /var/tmp/upgrade.err /var/tmp/upgrade.out
luupgrade -u -n s10u6 /
-l /var/tmp/upgrade.err -o /var/tmp/upgrade.out /
-s $CD -j /var/tmp/profile~95 min on V240
fix bogus permissions for zone pathes created by lu*
# umount all zone ZFSs created by lu*, e.g.:
zfs umount /pool1/zones/*-zfs1008BE-s10u6
# fix permissions
chmod 0700 /pool1/zones/*-zfs1008BE-s10u6Leave these zone dirs unmounted, otherwise lumount/luactivate and friends will fail!
check infos and errors and fix them if neccessary, re-apply your changes
lumount -n s10u6 /mnt
cd /mnt/var/sadm/system/data/
less upgrade_failed_pkgadds upgrade_cleanup locales_installed vfstab.unselected
cleanup4humans.sh /mntBTW: We use the script cleanup4humans.sh to get this job done faster and in a more reliable manner. It prepares basic command lines, which one probably needs to use to decide, whether to copy back replaced files or to replace files with the version suggested by the upgrade package.
apply new patches
We use PCA to find out, which patches are available/recommended for the new BE and its zones and finally apply them using luupgrade -t ...
However, to get reproducable results on all systems, we modified the script to have the patch download directory preset, to use always /usr/bin/perl no matter, how the PATH variable is set, and finally to invoke a postinstall script (if available on the system), which automatically fixes questionable changes made by patches. For your convinience, you can download this Patch and edit/adapt it to your needs.
Of course you may try to accomplish the same with the smpatch command, however IMHO this kind of bogus, poorly designed software is probably one reason, why some people think, Java is a bad thing. That's why we don't install the SUNWCpmgr, SUNWCswup, SUNWswupclr, SUNWswupcl junkware on any system.
OK, let's start with creating patch lists for each zone using the convinience script lupatch.sh (download and adjust it to your needs if you don't have it already). BTW: This command works also for the current BE, when one adds the '-R /' option.
# download all potential patches and show available patch lists when finished
lupatch.sh -dNow one should study the READMEs of all potential new patches for the zones by invoking the command shown below. BTW: This command works also for the current BE, when one adds the '-R /' option.
lupatch.sh -r
After that one should customize the available patch lists (per default /var/tmp/patchList.*) by removing the patch list at all or by removing the lines of patches, which should not be installed.
Finally one should apply the patches to the zones either using 'luupgrade -t -r /mnt patch ...' or the following command, which uses 'pca -i -R /mnt/$zonepath patch ...' internally to patch each zone using the patch lists from mentioned before (and makes sure, that the global zone gets patched first).
lupatch.sh -i
Make sure to always patch the 'global' zone FIRST!!!
unmount the BE
cd /
luumount s10u6activate the new BE
luactivate s10u6
boot into the new BE
init 6
# after reboot, check whether you still have a swap device enabled
swap -l
# if not, add the one of from that pool, where the current BE lives and
# add an appropriate entry to /etc/vfstab:
swap -a /dev/zvol/dsk/rpool0/swap
echo '/dev/zvol/dsk/rpool0/swap - - swap - no -' >>/etc/vfstabadjust your backup settings
javaws -viewer
Cleanup - Phase 2
In this section it is shown, how one may transfer remaining ZFSs from HDD1 aka rpool with the old ZFS BE (zfs1008BE) to HDD0 aka rpool0 where the current BE (s10u6) lives. This is usually not neccessary, when one was able to attach s0 of HDD0 to s0 of HDD1 before doing the upgrade to the "new" OS. However, for some reasons this might have not been possible at the time of the upgrade and that's why this case including pretty detailed troubleshooting hints are covered here.
To get the picture, we have now the following situation on our server:
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
zfs1008BE yes no no yes -
s10u6 yes yes yes no -
Filesystem Size Used Available Capacity Mounted on
rpool0/ROOT/s10u6 33G 4.7G 23G 18% /
rpool0/ROOT/s10u6/var 33G 2.2G 23G 9% /var
rpool/export 34G 20K 21G 1% /export
rpool/export/scratch 34G 1021M 21G 5% /export/scratch
pool1 67G 22K 66G 1% /pool1
pool1/home 67G 39M 66G 1% /pool1/home
pool1/web 67G 27K 66G 1% /pool1/web
pool1/web/iws2 67G 284K 66G 1% /pool1/web/iws2
pool1/web/iws2/sites 67G 850K 66G 1% /pool1/web/iws2/sites
pool1/web/theo2 67G 25K 66G 1% /pool1/web/theo2
pool1/web/theo2/sites 67G 24K 66G 1% /pool1/web/theo2/sites
pool1/zones 67G 29K 66G 1% /pool1/zones
rpool 34G 21K 21G 1% /rpool
rpool/var 34G 19K 21G 1% /rpool/var
rpool/var/log 34G 18K 21G 1% /rpool/var/log
rpool0 33G 21K 23G 1% /rpool0
rpool0/ROOT 33G 18K 23G 1% /rpool0/ROOT
rpool/var/log/web 34G 1012K 21G 1% /var/log/web
pool1/zones/sdev-zfs1008BE-s10u6 67G 732M 66G 2% /pool1/zones/sdev-zfs1008BE-s10u6
NAME USED AVAIL REFER MOUNTPOINT
pool1 1.10G 65.8G 22K /pool1
pool1/home 39.3M 65.8G 39.1M /pool1/home
pool1/home@home 196K - 39.1M -
pool1/web 1.18M 65.8G 27.5K /pool1/web
pool1/web/iws2 1.11M 65.8G 284K /pool1/web/iws2
pool1/web/iws2/sites 850K 65.8G 850K /pool1/web/iws2/sites
pool1/web/theo2 50K 65.8G 25.5K /pool1/web/theo2
pool1/web/theo2/sites 24.5K 65.8G 24.5K /pool1/web/theo2/sites
pool1/zones 1.06G 65.8G 29K /pool1/zones
pool1/zones/sdev 942M 65.8G 942M /pool1/zones/sdev
pool1/zones/sdev@zfs1008BE 264K - 942M -
pool1/zones/sdev-zfs1008BE 35.5M 65.8G 943M /pool1/zones/sdev-zfs1008BE
pool1/zones/sdev-zfs1008BE@s10u6 7.00M - 942M -
pool1/zones/sdev-zfs1008BE-s10u6 104M 65.8G 732M /pool1/zones/sdev-zfs1008BE-s10u6
rpool 12.5G 21.2G 21K /rpool
rpool/ROOT 7.52G 21.2G 18K legacy
rpool/ROOT/zfs1008BE 7.52G 21.2G 4.05G /.alt.zfs1008BE
rpool/ROOT/zfs1008BE/var 3.47G 21.2G 3.47G /.alt.zfs1008BE/var
rpool/dump 2.01G 21.2G 2.01G -
rpool/export 1021M 21.2G 20K /export
rpool/export/scratch 1021M 21.2G 1021M /export/scratch
rpool/swap 2.00G 23.2G 15.7M -
rpool/var 1.02M 21.2G 19K /rpool/var
rpool/var/log 1.01M 21.2G 18K /rpool/var/log
rpool/var/log/web 1012K 21.2G 1012K /var/log/web
rpool0 10.9G 22.6G 21.5K /rpool0
rpool0/ROOT 6.89G 22.6G 18K /rpool0/ROOT
rpool0/ROOT/s10u6 6.89G 22.6G 4.72G /
rpool0/ROOT/s10u6/var 2.17G 22.6G 2.17G /var
rpool0/dump 2.01G 22.6G 2.01G -
rpool0/swap 2.00G 24.6G 16K -
Dump content: kernel pages
Dump device: /dev/zvol/dsk/rpool0/dump (dedicated)
Savecore directory: /var/crash/joker
Savecore enabled: yes
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool0/swap 256,4 16 4196336 4196336
mount the new bootenv on /mnt - fix any errors
This step is recommended, since if lumount fails, ludelete would fail as well. So fixing it early makes troubleshooting easier.
lumount -n zfs1008BE /mnt
And ooops, we get the following error:
ERROR: cannot mount '/mnt/var': directory is not empty
ERROR: cannot mount mount point device
ERROR: failed to mount file systemon
ERROR: unmounting partially mounted boot environment file systems
ERROR: No such file or directory: error unmounting
ERROR: umount: warning: rpool/ROOT/zfs1008BE not in mnttab
umount: rpool/ROOT/zfs1008BE no such file or directory
ERROR: cannot unmount
ERROR: cannot mount boot environment by nameHmmm, when we 'ls -al' /mnt, the directory is empty. Since we are a little bit lazy to think about what's going on here, don't know what lumount internally does and do not want to speculate around, we simply open another terminal and use the following "oneliner" for checking:
dtrace -n /
'proc:::exec-success { printf("%d/t%s/n", timestamp, curpsinfo->pr_psargs); }' /
> ~/tmp/lumount.trcand execute the previous lumount command again. When it is finished we stop dtrace with ^C. Now we sort the output by timestamps (removing the overhead) and have a look at it:
gawk -f /local/misc/etc/tsortd.gawk ~/tmp/lumount.trc
CPU PID TIME COMMAND
1 48923 12341946782479 + lumount -n zfs1008BE /mnt
1 48923 12341952594063 lumount -n zfs1008BE /mnt
0 48923 12341971394229 /etc/lib/lu/plugins/lupi_zones plugin
1 48923 12341972471729 /etc/lib/lu/plugins/lupi_svmio plugin
0 48923 12341991109479 /etc/lib/lu/plugins/lupi_bebasic plugin
1 48923 12342011131729 metadb
0 48923 12342087965979 zfs set mountpoint=/mnt rpool/ROOT/zfs1008BE
1 48923 12343091202978 zfs get -Ho value mounted rpool/ROOT/zfs1008BE
1 48923 12343115644562 zfs mount rpool/ROOT/zfs1008BE
1 48923 12343186977645 zfs get -Ho value mounted rpool/ROOT/zfs1008BE/var
0 48923 12343212715811 zfs mount rpool/ROOT/zfs1008BE/var
0 48923 12344023001311 lockfs -f /mnt/
0 48923 12344027045227 umount -f /mnt/
0 48923 12344031500477 umount -f /mnt
1 48923 12344561966727 umount rpool/ROOT/zfs1008BE
1 48923 12344568052643 umount -f rpool/ROOT/zfs1008BENow it is clear: lumount does "nothing special" or "magic things". Mounting the the old BEs / succeeds, however mounting its /var FS fails. because the mountpoint of the BEs / has been already set to /mnt by lumount, we do:
zfs mount rpool/ROOT/zfs1008BE
ls -alR /mnt/var/mnt/var:
total 14
drwxr-xr-x 4 root root 4 Nov 27 05:09 .
drwxr-xr-x 33 root root 35 Dec 2 03:41 ..
drwx------ 3 root root 3 Nov 27 05:09 log
drwx------ 2 root root 4 Nov 27 05:30 run
/mnt/var/log:
total 9
drwx------ 3 root root 3 Nov 27 05:09 .
drwxr-xr-x 4 root root 4 Nov 27 05:09 ..
drwx------ 2 root root 2 Nov 27 05:09 web
/mnt/var/log/web:
total 6
drwx------ 2 root root 2 Nov 27 05:09 .
drwx------ 3 root root 3 Nov 27 05:09 ..
/mnt/var/run:
total 10
drwx------ 2 root root 4 Nov 27 05:30 .
drwxr-xr-x 4 root root 4 Nov 27 05:09 ..
-rw-r--r-- 1 root root 4 Nov 29 06:30 bootadm.lock
-rw-r--r-- 1 root root 3 Nov 29 06:30 ipmon.pidBecause /var/run is usually mounted on swap and all other directories are empty, we can conclude, that we probably see relicts from the upgrade process but don't really care about it, because not needed anymore. So finally, we solve our problem and check again using the commands shown below.
rm -rf /mnt/var/*
zfs umount /mnt
lumount -n zfs1008BE /mnt
luumount zfs1008BEWell, now all this works and we are able to continue with the next step.
delete the old BE
Never delete a BE, where an error occured during lucreate when snapshoting relevant datasets. E.g.
Creating snapshot for
on /
ERROR: cannot create snapshot /
'pool1/zones/sdev-zfs1008BE-s10u6@s10u6_20081203': dataset is busy
ERROR: Unable to snapshoton /
cannot open 'pool1/zones/sdev-zfs1008BE-s10u6_20081203': dataset does not exist
cannot open 'pool1/zones/sdev-zfs1008BE-s10u6_20081203': dataset does not exist
cannot open 'pool1/zones/sdev-zfs1008BE-s10u6_20081203': dataset does not exist
Population of boot environmentsuccessful. First create another snapshot of the original ZFS (here pool1/zones/sdev-zfs1008BE-s10u6@stillused) to prevent ludelete from destroying it (here pool1/zones/sdev-zfs1008BE-s10u6). It's probably a bug and should not happen, however, it happend - the whole zone from the current BE was gone as well (even so not reproducable)!
ludelete zfs1008BE
lustatusumount: warning: /.alt.tmp.b-c0.mnt/pool1/zones/sdev-zfs1008BE not in mnttab
umount: /.alt.tmp.b-c0.mnt/pool1/zones/sdev-zfs1008BE not mounted
Deleting ZFS dataset.
ERROR: cannot destroy 'pool1/zones/sdev-zfs1008BE': filesystem has dependent clones
use '-R' to destroy the following datasets:
pool1/zones/sdev-zfs1008BE-s10u6
ERROR: Unable to delete ZFS dataset.
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environmentdeleted.
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
s10u6 yes yes yes no -As we can see, the old zfs1008BE with its ZFSs rpool/ROOT/zfs1008BE, rpool/ROOT/zfs1008BE/var, pool1/zones/sdev-zfs1008BE and pool1/zones/sdev@zfs1008BE could be deleted successfully, but not without any "errors". Of course we do not want to have pool1/zones/sdev-zfs1008BE-s10u6 destroyed, since it is still used as the base of the zone sdev. So
DON'T do, what ludelete actually suggests!What we do instead is to promote this ZFS clone of the pool1/zones/sdev@zfs1008BE ZFS snapshot (see zfs(1M)). Note, that this is not really required (because on pool1), but since the original BE doesn't exist anymore and the old zone (at least if it is a sparse zone) would use ZFS from the new BE, a rollback to the old zone would not make sense, because then it would be more or less inconsistent and cause sooner or later trouble. So free its resources aka disk space!
promote zone ZFSs, which depend on the old BE
# check potential candidates
zoneadm list -cp | grep -v :global: | cut -f2,4 -d:
# promote apropriate ZFSs
zfs promote pool1/zones/*-zfs1008BE-s10u6
# make sure, the VALUE of the origin is '-'
zfs get origin pool1/zones/*-zfs1008BE-s10u6destroy snapshots and old parents of promoted ZFSs
zfs destroy pool1/zones/sdev-zfs1008BE-s10u6@zfs1008BE
zfs destroy pool1/zones/sdevtransfer remaining ZFSs
So what is left?
zfs list | egrep '^rpool( |/)'
ksh
zoneadm list -cp | grep -v :global: | cut -f2 -d: | while read ZN ; do
echo $ZN
zonecfg -z $ZN info fs | /usr/xpg4/bin/egrep -E '[[:space:]](dir|special):'
zonecfg -z $ZN info zonepath | grep ': rpool/'
done
exit
# not shown below
swap -l
dumpadm
ls -al /rpool /export /rpool/var /rpool/var/logrpool 5.01G 28.7G 21K /rpool
rpool/ROOT 18K 28.7G 18K legacy
rpool/dump 2.01G 28.7G 2.01G -
rpool/export 1021M 28.7G 20K /export
rpool/export/scratch 1021M 28.7G 1021M /export/scratch
rpool/swap 2.00G 30.7G 15.7M -
rpool/var 1.02M 28.7G 19K /rpool/var
rpool/var/log 1.01M 28.7G 18K /rpool/var/log
rpool/var/log/web 1012K 28.7G 1012K /var/log/web
sdev
dir: /var/log/httpd
special: /var/log/web/iws2
dir: /data/sites
special: /pool1/web/iws2/sitesSo we can see, that the only ZFS we have to consider for transfer are rpool/var/log/web and rpool/export/scratch, because all others are either not used anymore or have no data in it (except the directory used as mountpoint for another ZFS). For obvious reasons we do not transfer /export/scratch but just destroy it and create a new one in rpool0. So /var/log/web is the only one left over.
However, since it is used by the zone sdev, we have to shutdown the zone first, transfer the fs and finally restart the zone.
# get rid of the old /export/scratch and create a new one on rpool0
zfs destroy -rR rpool/export
zfs create -o mountpoint=/export rpool0/export
zfs create -o sharenfs='rw=sol:bsd:lnx,root=admhosts' /
-o quota=8G pool0/export/scratch
chmod 1777 /export/scratch
# transfer /var/log/web
zlogin sdev 'init 5'
sh -c 'while [ -n "`zoneadm list | grep sdev`" ]; do sleep 1; done'
zfs umount rpool/var/log/web
zfs snapshot rpool/var/log/web@xfer
zfs send -R rpool/var/log/web@xfer | zfs receive -d rpool0
# restart the zone
zoneadm -z sdev boot
# fix /etc/power.conf
gsed -i 's,/rpool/,/rpool0/,g' /etc/power.conf
pmconfig
# optional: destroy the source of transfered ZFS
zfs destroy rpool0/var/log/web@xfer
zfs destroy -r rpool/var
zfs destroy -r rpool/exportSince the new ZFS inherits all properties from the old one (caused by the -R option), changing its mountpoint back to /var/log/web etc. is not needed - zfs will even mount it automatically on /var/log/web as long as this directory is empty and not in use.destroy the old pool
Just to make sure, that not unintentionally valid data gets destroyed, one should have a look at the pool!
zfs list | egrep '^rpool( |/)'
zpool destroy rpoolrpool 4.01G 29.7G 20K /rpool
rpool/ROOT 18K 29.7G 18K legacy
rpool/dump 2.01G 29.7G 2.01G -
rpool/swap 2.00G 31.7G 15.7M -make lufslist and friends happy
Since there is no lusyncicf one needs to do this manually. The helper below should make and display the proper changes, however one should of course verify, that the changes being made are correct.
ksh
BE=`lucurr`
ICF=`grep :${BE}: /etc/lutab | awk -F: '{ print "/etc/lu/ICF." $1 }'`
cp -p $ICF ${ICF}.bak
gsed -i -r -e '/:rpool:/ d' -e 's,:(/?rpool)([:/]),:/10/2,g' $ICF
diff -u ${ICF}.bak $ICF
print "/nVERIFY that $ICF is correct - if not/ncp -p ${ICF}.bak $ICF/n"
lufslist $BE
exitformat HDD1 and zpool attach to HDD0
Since we want to the boot disk mirrored by ZFS, we have to make sure, that s0 of HDD1 has at least the same size in blocks as s0 of HDD0. Otherwise zfs will deny attaching it.
# compare 'Sector Count' of 'Partition' 0
prtvtoc /dev/rdsk/c0t0d0s2 | /usr/xpg4/bin/grep -E 'Count|^[[:space:]]*0'
prtvtoc /dev/rdsk/c0t1d0s2 | /usr/xpg4/bin/grep -E 'Count|^[[:space:]]*0'
# if size is not OK, adjust it via 'format' - see above
# and finally attach the slice to form a 2-way mirror
zpool attach rpool0 c0t0d0s0 c0t1d0s0
zpool status
sleep 900
print "/n##### status #####/n"
zpool status | grep scrub:* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 71763164 71763163
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 72169752 72169751
pool: pool1
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
pool1 ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
errors: No known data errors
pool: rpool0
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h0m, 3.62% done, 0h23m to go
config:
NAME STATE READ WRITE CKSUM
rpool0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0
c0t1d0s0 ONLINE 0 0 0
errors: No known data errors
##### status #####
scrub: none requested
scrub: resilver completed after 0h14m with 0 errors on Wed Dec 3 06:05:06 2008Resolve ZFS Boot Issues
see ZFS Boot Issues and Solaris 10 10/08 Release and Installation Collection >> Solaris 10 10/08 Release Notes >> 2. Solaris Runtime Issues >> File Systems.
# ------------ on sparc ---------
dd if=/dev/rdsk/c0t1d0s0 of=/tmp/bb bs=1b iseek=1 count=15
dd if=/dev/rdsk/c0t1d0s0 of=/tmp/bb bs=1b iseek=1024 oseek=15 count=16
cmp /tmp/bb /usr/platform/`uname -i`/lib/fs/zfs/bootblk
# if they differ, fix it:
installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /
/dev/rdsk/c0t1d0s0
ls -al /rpool0/platform/`uname -m`/bootlst
# if this failed
mkdir -p /rpool0/platform/`uname -m`/
cp -p /platform/`uname -m`/bootlst /rpool0/platform/`uname -m`/bootlst
# ------------ on x86 (not yet tested) -----------
installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t1d0s0cleanup artifacts and have a beer
rm -rf /.alt.*
rmdir /rpool
day-to-day patching (ZFS)
When you have studied and understood all previous sections, you should have a pretty good understanding, how to do the day-to-day patching of a ZFS based Solaris system. However, for our convinience the summary.
+ tcsh
mount /local/misc
set path = ( /usr/bin /usr/sbin /local/misc/sbin )
setenv RPOOL rpool0
setenv NBE s10u6_`date '+%Y%m%d'`
lupatch.sh -d -R /
# stop here - if no new patches are available
lupatch.sh -r -R /
# edit patch lists for each zones if not all uninstalled patches should be
# installed:
# delete all unwanted patch lines - for permanent ignorance put something
# like 'ignore=123456-07' into /etc/pca.conf
vi /var/tmp/patchList.*
lucreate -n $NBE -p $RPOOL
umount /mnt
lumount -n $NBE /mnt
# do not continue before lumount above works - see lumount trouble
# Usually the zonepathes/mountpoints in the new BE are not set correctly
# or lucreate left some artifacts (directories) beneath ZFS mountpoints.
# To get rid of the former one permanently, you can try the luclonefs.patch
lupatch.sh -i
cd /
luumount $NBE
luactivate $NBE
# when the next maintainance window starts
init 6