Dell VxRail

Hyperconverged systems are a set of physical servers running in a cluster that provides compute, storage and networking all in one.  Over the past few years, hyperconverged system have been on the rise in popularity due to the highly scalable and ease of management with businesses.  It is backed by VMWare's vSAN technology.  IT administrators like it because these features are delivered by a single vendor but more importantly, the upgrade process is easy without much thought needing to research for firmware vs software and driver compatibility.  To scale the hyperconverged system (compute or storage), you can just add a new node/server to the cluster.

The VxRail is a hyper-converged appliance from Dell/EMC.  There are different versions of VxRails depending on your need for cpu, memory and size/number of disks.  The underlying hardware architecture are just their PowerEdge line of servers.  On the network side, it supports configurations of 1gb, 10gb or 25gb connecting to two top of rack switches for redundancy.  On the storage side, you can either go all Flash or Hybrid of SSD and HDD's.  But in order to do the minimum of FTT=1, at least 3 host is required.

VxRail Node Comparisons
	G Series	E Series	V Series	P Series	S Series
Form Factor	2U4N	1U1N	2U1N	2U1N	2U1N
Cores	4 - 56	4 - 56	8 - 56	8 - 56	4 - 56
Memory	64 GB - 2048 GB	64 GB - 3072 GB	128 GB - 3072 GB	64 GB - 3072 GB	64 GB - 3072 GB
Hybrid Storage Capacity	1.2 TB - 12 TB SAS	1.2 TB - 19.2 TB SAS	1.2 TB - 48 TB SAS	1.2 TB - 48 TB SAS	4 TB - 48 TB SAS
All-Flash Storage Capacity	1.92 TB - 19.2 TB SAS or 1.92 TB - 19.2 TB SATA	1.92 TB - 30.7 TB SAS or 1.92 TB - 30.7 TB SATA	1.92 TB - 76.8 TB SAS or 1.92 TB - 76.8 TB SATA	1.92 TB - 76.8 TB SAS or 1.92 TB - 76.8 TB SATA	Hybrid only
Use Cases	General-purpose for broad hyper-converged use cases	Basic for remote office, stretch cluster or entry workloads	Graphics-ready for uses such as high-end 2D/3D visualization	High-performance optimized for heavy workloads such as databases	Capacity-optimized with expanded storage for collaboration, data, and analytics

Deployment:

The deployment is done by Dell engineers with their Pro Deploy or Pro Deploy Plus package.  I don't believe you can purchase it without those packages.  On the technical side, the engineer will run their network assessment tool to determine if everything is accessible.  Any issues that come up on the network assessment tool will generally need to be resolved before moving forward with the install.  If everything passes, the idrac ip will be set up via a console connection for each host in your cluster by using the racadm tool.  Logging into the console on the first host, the management IP is configured and the VxRail Manager VM will be powered on.  Once the VxRail Manager is powered on, the engineer will go through the GUI which will ask a series of questions concerning IP, hostnames and passwords to your vCenter environment.  From that information, the rest of the hosts in your cluster will be deployed and configured automatically.

Licensing and Software:

-With each VxRail purchase, you get the following:

• VxRail Manager
• ESRS
• VMWare vCenter
• VMWare Log Insight
• EMC RecoverPoint

Upgrade process:

The upgrade process is rather simple.  This is done from the VxRail Manager.  You can either do an upgrade via the internet or a local upgrade for those who have slow internet access.  This is will download the new VxRail package, unzip it and prep it for upgrading.  Upgrading, of course, is not intrusive to the VM's in the cluster.  Each host is upgraded one at a time while your VM's vMotions off to another host during the process.  From my experience, each host takes about 40mins to upgrade depending what level of a jump you're moving to.

How to check what services are running in VCSA.

SSH into your VCSA and enter the shell.  Type in the following command.

service-control --status --all

To start a service type in the following command.

service-control --start <service name>

VxRail iDRAC cold reset

If you're experiencing weird issues with your iDRAC(i.e. reporting false hardware failure information) you might want to give it a reboot. From VxRail, SSH into the ESX host and do the following.

      cd /opt/vxrail/tools/
      ipmitools mc reset

The iDRAC will get disconnected and will reboot. This generally takes 3-4 mins. The server and its services will remain running.

Cohesity: Backup, Replication and Disaster Recovery.

I had the opportunity to test another backup and DR solution recently.  This time, its Rubrik's competitor Cohesity.  If you miss the Rubrik overview, you can search for 'Rubrik' in the search field located on the right.  From the looks of the hardware (model C2300), it looks similar as the Rubrik device we've evaluated from a standpoint of each having 4 nodes blades with (12) 3.5" disks condensed into a 2U device but with a different faceplate.  After having looked at Rubrik, I was excited in finding out what Cohesity has to offer.  My initial thoughts, without actually diving into it, was that these two units does the same thing in the same way.  After playing with it for a couple of days, I was correct.  There are a lot of similarities between the two and some minor differences that does not have anything to do with what it advertise for.  They both do backups based on taking a vSphere level VM snapshot.  For replication, Cohesity copies the snapshotted VM over to a second appliance.  And for DR recovery, it would just bring up a copy of the VM that got replicated to the second device.  As with Rubrik, one of their main selling point is that your can bring up a DR instance within seconds by the virtue of mounting the Cohesity device to our ESXi host as a datastore and powering on the VM from there without actually having to do any restores.  One thing that Cohesity is different than Rubrik is that Cohesity is job and policy based whereas Rubrik is SLA based.  I much prefer Cohesity's job and policy bases over the SLA base because I can see all the VM's failed in one job in a single pane of glass view.  Can't do this with Rubrik unless your run a report.  Cohesity does offer you to create seperate storage domain so you can see how much a job uses because the reporting is storage domain based.  By doing so, you get less deduplication as it does not dedupe across other storage domains.  Some people wouldn't like it to have multiple storage domains but I find it the only way to see how much a space a particular job with Cohesity.  Like Rubrik, Cohesity is very scalable.  If you need more storage, you would purchase another node and add it to the Cohesity cluster.  Done.   To go further into the backup realm, Cohesity has the ability to archive backups to a third party cloud, able to backup up a network folder, and SQL database level backup using an agent.   We found that the cost for purchasing Cohesity was suprisingly low.  For a C3400, which provides a usable capacity of ~40TB (don't exactly remember the actual number), the cost was less than $90k for 2 units, with 3 year maintenance contract.

C2000 series

     

4 node cluster

Pros:

• Can do VM backups, replication, DR and archive.
• Surprisingly low cost.
• Storage domains.
• Easy to scale.
• Job and policy based.
• Very granular retention policy.
• Can archive to third party cloud.
• Can back up network shares and SQL databases.
• Active directory integration.
• Good detailed audit logging.
• Easy upgrade process.
• Can mount itself as an ESXi datastore and run VM's off it making RTO time very low.
• REST API

Cons:

• vSphere level snapshot for backups.  Can be problematic for VM's with large disks depending on what application it is running.  
• Hardware based.  Will need to recycle hardware when it gets old.
• RBAC permissions didn't work too well. 

Reset Local Administrator Password (Windows 10)

If you've run into a situation where you can't log into a Windows 10 computer because your domain credentials no longer work as the computer object was deleted from Active Directory and you don't know the local Administrator account.  You can reset the local Administrator account using the steps below.

• Hold down shift and click on the reboot button on the Windows 10 login screen.  It will reboot into recovery mode.
• Go into the Command Prompt from recovery mode.
• Enter in the Bitlocker Key if your disk is Bitlocked.
• Overwrite utilman.exe with cmd.exe.
• move c:\windows\system32\utilman.exe c:\windows\system32\utilman.exe.bak
• copy c:\windows\system32\cmd.exe c:\windows\system32\utilman.exe
• Reboot the computer using the following command.
• wpeutil reboot
• On the login prompt after rebooting, click on the Utility Manager icon.  This will open up the command prompt.
• Change the password of the Adminstrator account by using the following command.
• net user administrator <new password>
• Exit the Command Prompt and log in with the new password.

Test your hard disk drive performance in Linux

Below are some usage of the hdparm tool in Linux.  Replace sda with your device name.

• Get Hard Disk information
• sudo hdparm -I /dev/sda

• Read Performance Test
• sudo hdparm -t /dev/sda

• Enable Read-Ahead
• sudo hdparm -A 1 /dev/sda

• Turn on DMA
• sudo hdparm -d1 /dev/sda

• Retain hdparm settings after reset
• sudo hdparm -K 1 /dev/sda

Run Powershell commands remotely with PSSession.

Here's a way to run Powershell commands on a remote machine.  It saves the hassle of RDP'ing into the remote machine, wait for Windows to load your profile, run startup scripts, apply GPO settings, load your desktop, etc...  That can be time consuming if you're always logging into to remote machine to verify something.

1. Create a remote session
• new-pssession <computer hostname>
2. Get the ID of the remote session.
• get-pssession
3. Connect to that remote session using the ID.
• enter-pssession <ID>
4. Once connected, you should see the computer hostname in brackets like below.
• [computername]: PS C:\users\guest
5. Execute your Powershell command like so.
• [computername]: PS C:\users\guest> get-volume
6. To exit the session, just type the following.
• exit or exit-pssession
7. After exiting of the session, you would want to remove/close the session from your computer.
• remove-pssession <id>
• Leave me a comment if you know of a way to remove 'all' the sessions at once without looping it in a script.

Hope this helps.

Rubrik: Backup, Replication and Disaster Recovery.

Just started looking into Rubrik to backup and DR my work's VM.  In short, Rubrik is a hardware appliance that does VM snapshot backup, replication and DR.  It's fairly straight forward to use and the web console is easy to navigate.  It does replication to a second Rubrik cluster for DR.  So if you want to do replication, you will need 2 appliances. One of their selling point is the Live Mount Recovery feature.  If you're in a scenario to recover a corrupted production VM, Live Mount Recover can mount the Rubrik device as a datastore on your ESX host and power on the VM directly.  This will provided you with minimal downtime from the traditional VM restore.  It also backs up physical machines granted you install their agent.  The agent will allow you to pick and choose what files you want to back up, but the downside to that is this agent backs up on a file level.  So no System State backups or bare metal restores.  Rubrik uses a SLA policy driven to do backups and archive.  You can create as many SLA policies you want and within each of those policies, you can specify when to do snapshots, how often to do it and how many snapshots to retain.   It's very granular in terms of snapshot flexibility.  Since Rubrik is a bit new to the scene, there are some functionality that it does not provide.  I'm hoping they are working on adding these features in for future releases.   Below are some of the pro's and con's.  Note that some of the current missing feature sets mentioned here might be available on the latest release or by the time you've read this.  But my recommendations before you jump into Rubrik is to also look at their direct competitor.

Rubrik device mounted on the rack.  This 2U 'Brik' has 4 nodes.

Front Bezel Removed.

Pros

•    Live mount recovery/Instant Recovery.  Can automatically mount the Rubrik device as a datastore in ESX and run VM's off it.
•     VM backups and replication
•     SQL db backups (with an agent installed)
•     Very granular SLA's and retention
•     Archive to cloud from SLA's
•     Granular file/search and recovery within a VM
•     Scalable.  You can add more nodes to the cluster for more space capacity.
•     End to end encryption (at rest and in transit)
•     Good granular reporting.
•     RBAC permissions.
•     Hardware and Software solution.  Nothing else you need to buy.
•     Deduplication

Cons

•      No active directory object level backups
•      No system state backups.
•     Web console difficult to navigate.  We found its difficult to see if a job has failed without doing a report.  They tend to want you to search by VM to determine if a VM has failed to back up.
•      Weak audit logging.  No way to tell if someone modified a job.  
•     Pricey compared to its competitors, maybe 2x pricey.  Mostly software and maintenance cost.
•      vSphere level snapshot backups.  Can be problematic if you have a vm with a large virtual disk as large snapshots can take time creating snapshot and cleaning up snapshots.

How to get network statistics on a VM nic from SSH.

If you need to get the network statistics via SSH on a VM nic, run the commands below.

1. 'esxcli network vm list' to get VM ID.
2. 'esxcli network vm port list -w <VM ID>' to get port ID number.
3. 'esxcli network port stats get -p <port ID number>' to get network statistics.

Output:

Packet statistics for port 50332931

   Packets received: 293182742829

   Packets sent: 90336672427

   Bytes received: 305397539251

   Bytes sent: 11805570153

   Broadcast packets received: 135893129

   Broadcast packets sent: 813717

   Multicast packets received: 386654045

   Multicast packets sent: 334554

   Unicast packets received: 292660195655

   Unicast packets sent: 90335524156

   Receive packets dropped: 0

   Transmit packets dropped: 0

4. 
   

Get Active Directory User's password expiration date.

Most corporation that uses Microsoft's Active Directory has set some password policies for tighter security.  One of them is probably 'Password Expiration' to force users to change their passwords every x amount of days.  I was in a situation where I needed to find a user account password expiration and the first thing that come to my mind is to use Powershell.  Upon Googling on how to do it with Powershell, I found that it is very cumbersome to do it.  You run a get-aduser command and it will output the accountexpires attribute but its not in datetime format.  You'd have to convert it, which didn't work for me.  Then I find that you can use the basic net user command like below.

c:\net user <username> /domain

That's all.