Category Archives: vCenter
Running Docker inside of VCSA 6.7
During this years VMware {code} Hackathon at VMworld Vegas I submitted a team project which ran PowerShell within the vSphere Client. To achieve this I had to create a Docker container running PowerShell Core. During development I ran the container on an Ubuntu Linux server. For my final submission at the Hackathon I wanted to minimise the dependency to run a separate Linux VM. So I created a process to install and run Docker directly on VCSA 6.7.
The process to install and run Docker within VCSA 6.7 is surprisingly very simple. As a reference I used a blog post from William Lam but with a small modification to correctly load the Bridge module in VCSA 6.7 (as opposed to VCSA 6.5 in William’s post).
I thought I would share the steps I used below for others to experiment with.
Step 1.
SSH to the VCSA VM and enter the Shell
Install the docker package
tdnf -y install docker
Step 2.
Load the kernel module to start the Docker client. (This step needs to be re-run if the VCSA is rebooted)
modprobe bridge --ignore-install
If successful no information should be returned.
You can check if the module is installed by running the following
lsmod | grep bridge
root@vc67-1 [ ~ ]# lsmod | grep bridge
bridge 118784 0
stp 16384 1 bridge
llc 16384 2 stp,bridge
Step 3.
Enable and start the Docker Clientsystemctl enable docker
systemctl start docker
Making the Bridge module load after a reboot (Optonal)
As with William’s process to install and run Docker Step 2 needs to be re-run each time the VCSA is rebooted. The solution to make the Bridge module automatically load after a reboot happens to be exactly the same. I’ve included the specific commands to simplify the process.
Step 1.
Make a backup of the file we are going to modify.
cp /etc/modprobe.d/modprobe.conf /etc/modprobe.d/modprobe.conf.bak
Step 2.
Comment out the line that disables the Bridge module from loading
sed -i "s/install bridge/# install bridge/g" /etc/modprobe.d/modprobe.conf
Step 3.
Create a new config file to load the Bridge module
touch /etc/modules-load.d/bridge.conf
Step 4.
Specify the name of the Bridge module to load at reboot
echo bridge >> /etc/modules-load.d/bridge.conf
That’s all we need to do to start running containers on VCSA 6.7. Excluding the steps to have the Bridge module persist after a reboot and it’s even simpler.
Running a Docker container
If you want to test your first container on VCSA you can try my Docker image I build for the Hackathon. The steps are very simple and listed out below.
Step 1.
Download the two required files from my GitHub repo
curl -O https://raw.githubusercontent.com/originaluko/vSphere_PowerCLI/master/Dockerfile
curl -O https://raw.githubusercontent.com/originaluko/vSphere_PowerCLI/master/supervisord.conf
Step 2.
Build the Image
docker build -t vsphere-powercli .
Step 3.
Start the container with this image
docker run -d -p 4200:4200 --name vsphere-powercli vsphere-powercli
You can check if the docker container is running with the below command.
docker stats
You can check if the port has been mapped correctly by running
docker port vsphere-powercli
Step 4.
Open a web browser and test if you can connect directly to the container through the browser and login to the PowerShell prompt.
https://{my_vcsa}:4200
Accept any certificate warning and login with the default credentials of powercli / password
I hope that was all fairly straight forward to understand. It’s actually all very simple to achieve under VCSA 6.7. As in William’s post, none of this is supported by VMware, so user beware. Though I have been playing with Docker inside of VCSA 6.7 for quite a few months now with no noticeable issues.
If you want to see my full VMworld Hackathon project you can check it out over in my GitHub repo
VMware Update Manager (VUM) Fails To Load Within vSphere Web Client
I recently upgraded my lab VCSA from version 6.5 (Build Number 5705665) to version 6.5 U1 (Build Number 6671409). After the upgrade I noticed that VMware Update Manager was no longer working correctly. Navigating around the various VUM pages I received the same consistent error message.
interface com.vmware.vim.binding.integrity.VcIntegrity is not visible from class loader
VUM management page
VUM Tab within an ESXi host
Checking the vCenter services within Administration > System Configuration they all appeared Up and Running. Though all services were running I never the less restarted the VMware Update Manager service which unfortunately didn’t help. I also tried restarting a few other services without much success. So rather than just continuing to randomly restart services I decided to take a tougher approach and restart all services from the CLI.
After the stopping and starting of all vCenter services, which took a few minutes, VUM was back up and running again within the vSphere Web Client. While this was a fairly drastic step to take, so would have been rebooting the vCenter server, which I’m glad I managed to avoid.
I’ve previous written about restarting vCenter services. The process is quite simple. First connect up to the CLI of the VCSA box. Then run the below two commands. Both the stopping and starting of services will take a few minutes each. Once the services are restart the Web Client will take a further few minutes to fully start up and be accessible. If all is successful Update Manager should be accessible once again.
Command> service-control --stop --all
Command> service-control --start --all
Restarting all the vCenter services like this is obviously a disruptive action. Connectivity to vCenter will be dropped while the services restart. Usually restarting all services on vCenter via the CLI is my last ditch attempt to resolve an issue before I attempt a reboot of the appliance. While restarting the VCSA might have been the easiest thing to do to resolve this issue it’s not always necessary.
Could not establish trust relationship for the SSL/TLS Secure Channel – Invoke-WebRequest
I’ve recently been playing around with VMware’s REST APIs in VCSA 6.5 using PowerShell. I’ve been using a lot of Invoke-WebRequest and Invoke-RestMethod to do my work. Chris Wahl has a great primer on how to get started here.
One issue that I ran into very quickly working again my VCSA was a certificate trust relationship error. I’ve run into this error numerous times in the past.
|
1
2
3
4
5
6
7
|
PS F:\Code>
Invoke-WebRequest
-Uri
https://10.0.0.201/rest/com/vmware/cis/session
-Method
Post
-Headers
$head
Invoke-WebRequest
: The underlying connection was closed: Could not establish trust relationship
for
the SSL/TLS secure channel.
At line:1 char:1
+
Invoke-WebRequest
-Uri
https://10.0.0.201/rest/com/vmware/cis/session ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [
Invoke-WebRequest
], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
|
The first time I ran into this error I was stumped for while finding a solution. Ultimately it comes down to using Self-Signed Certificates in vCenter, as most of us do. In general using Invoke-WebRequest or Invoke-RestMethod against a server using a Self-Signed Certificate will cause this error, it’s not just related to vCenter.
The solution is quite simple. I found a snippet of code some time back that I keep on hand in this situation. It basically ignores certificate validate in PowerShell allowing you to make a connection with Invoke-WebRequest. All you have to do it paste this code into your PowerShell session before you run Invoke-WebRequest against a server with a Self-Signed Certificate.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
if
(
-not
(
[System.Management.Automation.PSTypeName]
'ServerCertificateValidationCallback'
).Type)
{
$certCallback
=
@"
using System;
using System.Net;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
public class ServerCertificateValidationCallback
{
public static void Ignore()
{
if(ServicePointManager.ServerCertificateValidationCallback ==null)
{
ServicePointManager.ServerCertificateValidationCallback +=
delegate
(
Object obj,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors errors
)
{
return true;
};
}
}
}
"@
Add-Type
$certCallback
}
[ServerCertificateValidationCallback]
::Ignore()
|
Once you run the code you will be able to now successfully make a connection.
I’ve seen some simple one liner solutions for Self-Signed Certificates but none of them seemed to work for me. Whereas the above snippet of code has always worked. Obviously bypassing certificate validate is not something you want to run on a global scale in PowerShell but this code works great for your current session only.
If there is a simpler way to bypass certificate validation I’d love to hear it.
Streaming Datasets – PowerShell | PowerCLI | Power BI
A large part of my day is spent scripting in PowerShell, specifically with PowerCLI. One of the strongest areas of PowerCLI, obviously, is being able to retrieve information. It’s one of the key use cases, in my opinion, for using PowerCLI in a VMware environment, it’s ability to retrieve information for Capacity planning and reporting.
Recently I’ve been looking at how to consume all that information. You can obviously export it to a CSV, push it into a database, or something that I’ve been playing around with recently, stream it into Power BI. Now if you haven’t tried it out yet, PowerBI is an analytics service from Microsoft. At its core it’s a data warehouse for business intelligence. But putting all those fancy words aside, I use it to create fancy reports.
Exporting information out of a vCenter environment with PowerCLI is dead simple. I have dozens of scheduled tasks running all the time doing this. Where I’ve fallen down, is taking that information and trending it over time. This is where the Streaming Datasets functionality of Power BI comes in. Using PowerCLI I can get an Object and Value from vCenter and then Post that directly into Power BI, using their API, and have it instantly graphed in a report. I can then share that report out to anyone I want. Power BI lets me do this over and over, almost as fast as I can pull the information out of vCenter.
In the example below I show how to create a trend report over time that displays Total and Available Storage of a vCenter Cluster. Rather simple, I know, but can easily be adapted to show things like number of running VMs running, reserved resources used, etc, etc. The skies the limit really.
Before we do any scripting the first thing we do is log into Power BI. If you don’t have an account, don’t worry, the basic version is free. Hit the Sign Up link and make sure you select Power BI and not Power BI Desktop for Windows, we want the cloud version.
Once logged in we click on Streaming Datasets in the bottom left under the Datasets category. This is where we create our initial dataset schema so that it can accept streaming input. We click on ‘Add streaminig dataset’ in the top right.
Then select the source of data, which will be API and click next.
We give our New Streaming Dataset a name and define a few values. In this example we will define a Date, Total Storage, and Available Storage value, turn on Historic Data Analysis and click Create. Make note of your data type to the right of the value. Date is DateTime and the other two are Numbers.
We’ve now created our schema and are provided with a Push URL address and sample code in a few different formats (we want PowerShell). If you look carefully we are using an Invoke-RestMethod to Post to Power BI. This sample code has just provided us the template and hardest part of our PowerShell / PowerCLI script. Click over the code and copy / pasta it out to use in our script (Paste it at the bottom of the script as it will be the last thing that runs).
Now we actually start on the PowerShell / PowerCLI script. To keep it as concise as possible. I’ve skip the process I use to actually connect to the vCenter and retrieve the information out using PowerCLI in the code below. The real goal here is just to retrieve some values and get that into Power BI. Line 6 is basically retrieving all shared VMFS datastores in Cluster1. The important lines to note, though, are 4, 8, and 9 where I store my key values in three variables. One for Date, one for TotalStorage, and one for AvailableStorage.
|
1
2
3
4
5
6
7
8
9
|
Import-Module
VMware.VimAutomation.Core
Connect-VIServer
-Server
host.mydomain.local
$date
=
Get-Date
$datastore
=
Get-Cluster
-Name
Cluster1 |
Get-Datastore
|
Where-Object
{
$_
.Type
-eq
'VMFS'
-and
$_
.Extensiondata.Summary.MultipleHostAccess}
$TotalStorage
= (
$datastore
|
Measure-Object
-Property
CapacityMB
-Sum
).Sum / 1024
$AvailableStorage
= (
$datastore
|
Measure-Object
-Property
FreeSpaceMB
-Sum
).Sum / 1024
|
The additional lines below from 11 onward is the important code. This is our pasted sample code from Power BI that we will slightly modify to push our values up to Power BI. Don’t copy mine, as your URL and key will be different. On lines 13, 14, and 15 we will remove the example values and replace it with our three variables, $Date, $TotalStorage, and $AvailableStorage.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
Import-Module
VMware.VimAutomation.Core
Connect-VIServer
-Server
10.1.1.201
-user
"mydomain\username"
$date
=
Get-Date
$datastore
=
Get-Cluster
-Name
Cluster1 |
Get-Datastore
|
Where-Object
{
$_
.Type
-eq
'VMFS'
-and
$_
.Extensiondata.Summary.MultipleHostAccess}
$TotalStorage
= (
$datastore
|
Measure-Object
-Property
CapacityMB
-Sum
).Sum / 1024
$AvailableStorage
= (
$datastore
|
Measure-Object
-Property
FreeSpaceMB
-Sum
).Sum / 1024
$endpoint
=
"https://api.powerbi.com/beta/83fe1fa2-fa52-4376-b7f0-cb645a5fcfced/datasets/d57970bc-60b3-46e6-b23b-d782431a72be/rows?key=2zEhgN9mu%2BEH%2FI2Cbk9hd2Kw4b5c84YaO6W8gzFcZbBnO6rti3N631Gjw%2FveNXSBxwR84VcWPGOSrheNwQnCbw%3D%3D"
$payload
= @{
"Date"
=
$Date
"Total Storage"
=
$TotalStorage
"Available Storage"
=
$AvailableStorage
}
Invoke-RestMethod
-Method
Post
-Uri
"$endpoint"
-Body
(
ConvertTo-Json
@(
$payload
))
Disconnect-VIServer
*
-Confirm
:
$false
|
On the last line I disconnect from my vCenter and close any sessions. This helps if running as a scheduled task. Finally save the script.
And that’s it for the scripting part. Assuming everything is correct, no connection issues, correct values being retrieved. All we have to do is run the script and it will send a POST request using Invoke-RestMethod with our three values. We can now run this script as many times as we want and it will continue to post the current date and time along with Total Storage and Available Storage. At this point, if we wish, we can turn the script into a scheduled task or just continue to run manually to suit our needs.
We now go back to Power BI and report on what we have. Back on our Streaming Datasets browser window we click the Create Report icon under actions. Now this part is going to be very subjective to the end user who wants the report. But the key data we want is under RealTimeData on the far right. Select all three values and we get presented with a default view of our data. Under Visualizations select line chart and now we start to see a more visual representation of our capacity over time. Under the Analyticssection add a trend line and see a basic view of available capacity over time. Finally hit save and you have a self updating report from streaming data.
For the report to start to look anything like below it will take time and a few sample datasets. In the below image I’ve mocked up some numbers over time as an example.
Once you have a working script and it’s streaming data to PowerBI it’s really up to you on how to report on it. The above example, as simple as it is, lays the ground work to more customized and complex reporting that you might not be able to get out of traditional monitoring and reporting software. The ability is there to even share out the report.
Streaming datasets, as you might have noticed in the UR, is still in beta. As great as I have found it to be it does have some quirks. For one you can’t easily modify data you have already streamed up to Power BI. So if you send incorrect data / values up to Power BI in a streaming dataset it will remain their. At which point you will have to consider Filters to exclude it in reports.
In summary I think Power BI is a very underrated free tool from Microsoft. I’ve only just started to scratch the surface of what’s possible with it. The simplicity of capturing data with PowerShell and sending it to Power BI is well worth the time and effort to try at least once. So what are you waiting for?
Modify HTML5 vSphere Client Idle Timeout
Before I go any further, just to make it clear, we’re talking about the new HTML5 client in vSphere 6.5 (GA Build 4602587). Not the older Flash based vSphere Web Client in vCenter 5 and 6. So lets call it the vSphere vCenter HTML5 UI Web Client. Clear now? Ok, just refer to the pic below.
Below are the steps I used on the vCenter Server Appliance.
Just like the old Web Client I know of no way to change the idle timeout from within the UI today. So we have to revert to connecting to the console and making the changes through the shell. We do this by opening up a console window to the VM or using SSH and login with root (remember to enable SSH first).
At the Command prompt of the VCSA type the following to enable Shell access. You may received a Shell is disabled message. If you do, enable with shell.set.
Command> shell
Shell is disabled.
Command> shell.set --enabled true
Command> shell
vc01:~ #
Now at the Shell type the following below and locate session.timeout.
cat /etc/vmware/vsphere-ui/webclient.properties
You should find something similar to session.timeout = 120 as this is the default value in minutes.
Make a backup copy of webclient.properties.
cp /etc/vmware/vsphere-ui/webclient.properties /etc/vmware/vsphere-ui/webclient.properties.bak
If you’re comfortable using an editor like VI go ahead and use that to increase or decrease the value in minutes. Probably for the best, it doesn’t appear that you can set this value to never timeout. I tried 0 and -1 and both caused the vSphere Client to timeout instantly on login. The timeout value, though, can quickly and easily be modified using the sed command.
The sed command below locates the specific string session.timeout = 120 and replaces it with session.timeout = 720, which is 12 hours (or in other words my standard work day). Change 720 to however many idle minutes you want. If sed doesn’t find the specific string, don’t worry, it won’t modify anything.
sed -i “s/session.timeout = 120/session.timeout = 720/g” /etc/vmware/vsphere-ui/webclient.properties
Run the cat command again and check that the session.timeout value has changed.
cat /etc/vmware/vsphere-ui/webclient.properties
If the session.timeout value has been modified correctly we now have to stop and restart the vsphere-ui service by running the following commands below. I covered stopping and starting all services on a VCSA in a previous post HERE.
service-control --stop vsphere-ui
service-control --start vsphere-ui
Wait a few minutes for the service to start up fully and open a new browser windows to the vSphere Client. It should now be running with a new idle timeout.
vCenter In VR (Is This VCSA 7?)
The last few months have been extremely fun for me. I purchased a HTC Vive and have been enjoying every minute with it. I’m not a huge gamer but I absolutely love the immersion factor. I’ve lost count of the times I have got lost in games like Onward for hours on end. The realism and social aspect of coordinating with your team mates on how to take the objective. The absolute fear of crouching behind a wall while the enemy next to you discusses where you are. An experience that’s hard to convey.
Games aside though, VR also has the ability to mirror your desktop and applications too. Nothing like Minority Report or that awesomely realistic movie Hackers. But think more a VR room with a computer screen in front of you that you can enlarge or shrink to suit your view.
So that got me thinking. There are a few different VR apps that let you mirror your desktop in VR. I decided to try out Bigscreen, mainly because it’s free! And hell, because this is a virtualization blog, I obviously had to try out the vSphere Client to see if I could practically manage my vCenter Homelab environment.
It took a few attempts to find the best viewing mode and way to manage vCenter with the vSphere Client. I first tried the large projector view on the wall in the VR room. This turned out to be an absolute joke. Imagine the worst, lowest, quality projector, and then try reading small text from the other side of a room. Then think of something worse. Okay… it wasn’t that bad but still.
Failing to use vCenter in the large projector mode view
The best mode I found was literally just sitting down in a chair. Switching to the floating screen mode and enlarging the screen to encompass my field of view. Then placing a small curve to the screen to rap a little around me.
Something’s red in my vCenter environment
I first tried managing vCenter with the HTC Vive controllers. The controllers basically act as laser pointers. You can pull up a virtual keyboard and laser zap the keys with the controllers as well as move the laser point around on the screen like a mouse cursor. Using projector mode this was okay but up close it was really awkward. Ultimately using the physical mouse and keyboard was most practical. And it was practical. As long as you can position your hands in the right spot and touch type there was no issues. You just have to adjust to what feels like a 100 inch screen in your face.
Bigscreen has what they call Mutliplayer rooms. This is where you can join and create a new room where people can share you screen experience. I did jump into some of these rooms where movies were playing and had a little chat to the other guests. I wasn’t game enough to create a room and share my vCenter screen though. I just felt that the VR community wouldn’t have the same appreciation for my vSphere Homelab environment
Jumping into someones VR cinema room
You can imagine how this multi-user room experience could be interesting though. Inviting a friend / service desk into your private VR room to help you out on an issue in your environment. Actually being able to point on the screen and talk through resolving an issue. Waving your hands in frustration when the service desk can’t fix your issue. It reminds me of the book Ready Player One. A dystopian future where lives are lived out in a VR world and virtual chat rooms.
So alright, all of this was a big gimmick. An excuse to talk about my HTC Vive and somehow justify it on my virtualization blog with vCenter. It was fun, though, I’m not holding my breath for vCenter 7 VR. But maybe a fling
SCP to a vCenter Server Appliance (VCSA)
For some this may be a rare situation but from time to time I find that I’m needing to copy files to and from a vCenter Server Appliance (VCSA). I had one of these situations recently on vCenter 6. I needed to move some log files off a VCSA box.
I’ve found the easiest way to do this is via SCP -- Secure Copy, which uses the SSH protocol. It’s a relatively simple process to enable the VCSA to accept SCP connections. It’s a two step process which first requires enabling SSH on the VCSA and then switching the default Shell.
Step 1, involves enabling SSH
I’ve written a previous post on how to enable SSH on a VCSA here. Since that post VMware have re-released the VAMI on vCenter Server Appliance V6 U2. So I thought I might show this new method to enable SSH. Only if using VCSA 6 U2 or greater else use my previous post steps.
Connect to the VAMI URL of your vCenter on port 5480 using HTTPS. In my case it was https://vc.ukoticland.local:5480/login.html
Login with your VCSA root account and password. Then navigate to Access and click Edit on the far right. Select Enable ssh login and to make life a little easier also Enable bash shell and click OK. The timeout refers to how long the Bash shell will stay enabled. The default is fine.
Step 2, changing the default shell
Even though we enabled the bash shell above the default shell is still the VMware appliance shell which prevents us from connecting to the VCSA via SCP. So we need to SSH to the VCSA and change the default Shell from the Appliance Shell to Bash.
In my case I used Putty. Logged in with my root account and type shell.
Now i can change the default shell for the root user to bash using the below command.
chsh -s /bin/bash root
We’re now ready to SCP to our VCSA with the ability to transfer files to and from the VCSA. I use the simple Windows app, WinSCP. I change the File Protocol to SCP. I enter in my vCenter as my host and my root credentials.
When you’re complete just reverse the changes you made. In the SSH Putty session type the below to permanently switch the Bash shell back to the default Appliance Shell. Then log back into the VAMI as above. In Access deselect SSH and Bash.
chsh -s /bin/appliancesh root