Information Systems:VMWare Production Infrastructure

From uniWIKI
Jump to navigation Jump to search

Hardware

uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

Hardware - Network Port Map

This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.

  • Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
  • Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
  • Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
  • Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
  • Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
  • Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
  • Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
  • Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
  • Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
  • Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

Hardware Purchase Details

The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

Hardware Support

uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

IMM2

The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

vSphere Hypervisor

The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.

  • Server with serial number J121W8C has a host name of vmhost01.unipharm.local
  • The administrator username is root
  • The administrator password is NewVisionIT
  • The management network is on eth2 which is the third network port on the back of the server
  • The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
  • The management network is using the default VLAN

and

  • Server with serial number J121W8D has a host name of vmhost02.unipharm.local
  • The administrator username is root
  • The administrator password is NewVisionIT
  • The management network is on eth2 which is the third network port on the back of the server
  • The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
  • The management network is using the default VLAN

The web administration pages to directly access the ESX hypervisor and bypass vCenter are:

The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

vCenter Server Appliance

During Stage 1 of the VCSA setup, the following settings were used

  • The FQDN of the VCSA is vcsa.unipharm.local
  • The IP address of the VCSA is 172.30.18.23
  • The password for the VCSA is NewVisionIT@2051
  • The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
  • The VCSA has an integrated, as in no external, Embedded Platform Services Controller
  • The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk

During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used

  • Single Sign-On domain name is vsphere.local
  • Single Sign-On user name is administrator
  • Single Sign-On password is NewVisionIT@2051
  • Site name is uniPHARM
  • SSH access was enabled
  • The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/

Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.

vCenter Management

The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

vCenter Management - vNetwork

As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:

  • The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
  • The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
  • The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
  • The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
  • The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
  • The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another

It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

VUM VMware Update Manager

VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

vCenter Alarm Configuration

vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:

  • "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
  • "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
  • "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
  • "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
  • "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
  • "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
  • "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

Content Library

Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

Virtual Machines

A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.

VCSA

The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

XClarity

XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :

The local login for the above URL is:

  • Username - adminit
  • Password - NewVisionIT2051
  • Active Directory SSO is currently not working in XClarity 1.4

The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.

  • It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
  • XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

Thermoprofile

This is the first physical server to be converted into a virtual machine:

  • Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
  • Configured the VM to use 1 vCPU and 8GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13

This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.

  • Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

Mail

This is the third physical server to be converted into a virtual machine:

  • The physical hard drive was thick provisioned and not shrunk
  • Configured the VM to use 2 vCPU and 8GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13

For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

SuperServer

This is the fourth physical server to be converted into a virtual machine:

  • The physical hard drives were thick provisioned and shrunk during the conversion process
  • The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
  • Configured the VM to use 2 vCPU and 8GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13

For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

WindowsXP JetDirect

DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

Lucy

This is the second physical server to be converted into a virtual machine:

  • Shrunk the 2 physical hard drives - C: from 150GB to 75GB
  • Configured the VM to use 2 vCPU and 4GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13
  • The VM does not have the serial ports that the physical machine had installed in its PCI slots

Smithers

This is the fifth physical server to be converted into a virtual machine:

  • The physical hard drive was thick provisioned and shrunk during the conversion process
  • The C: drive was configured to be 150GB
  • Configured the VM to use 2 vCPU and 16GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13

The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

WDS

This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.

  • The virtual hard drive was thin provisioned at 70GB
  • The C: drive was configured to be 100GB
  • Configured the VM to use 1 vCPU and 8GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13

UWDDC1

This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.

  • The virtual hard drive was thin provisioned at 50GB
  • The C: drive was configured to be 50GB
  • Configured the VM to use 1 vCPU and 4GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13

For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

UWDDC2

This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.

  • The virtual hard drive was thin provisioned at 50GB
  • The C: drive was configured to be 50GB
  • Configured the VM to use 1 vCPU and 4GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13

For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

XTGUI

This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.

  • The virtual hard drive was thick provisioned at 40GB
  • The C: drive was configured to be 40GB
  • Configured the VM to use 2 vCPU and 8GB of memory
  • Configured the VM to use a VMXNET3 and connect it to the Production LAN
  • The VM hardware level is 13

3CX Server Appliance

This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.

  • The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
  • The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
  • As of September 2018, this VM is not being backed up in any way
  • The virtual machine has the OpenVM Tools installed
  • The virtual machine is located on VMHost03
  • There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

Third Host Used For Testing

On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:

The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

VMware Tools

VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

vCenter Standalone Converter

The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

VMware Licensing And Support

uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at

  • www.vmware.com
  • Username is darrenf@unipharm.com
  • Password is visionit
  • Account number 114624681
  • Customer number 9027898369

uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

VM To Host Optimized Layout

thing

VMHost01 VMHost02 VMHost03 Purpose Fax number Username/login Mapped aliases Account number Notes
Administration Account Main Inbound/Outbound Primarily for admin purposes, also the ultimate route through which outbound faxes travel. 604-276-8478 admin@unipharm.com norwinu, darrenf, jeremym, nancyn 92531 nancyn and jeremym aliased to this account for address book management and billing administration, respectively.
Outbound Fax - Company Main Sub Outbound only Used to group users (majority of staff members) for outbound faxing 604-276-8478 (routes through Main account) fax@unipharm.com, or aliased users Most staff members, with their usernames in AD format (e.g. norwinu; Exceptions: tomc, timm1) 99797
Outbound Fax - Purchasing Dept. Sub Outbound only Used to group Purchasing Dept. users for outbound faxing 604-276-8478 (routes through Main account) purchasing@unipharm.com, or aliased users elizag, johnt, judyt, rubys, shannonm, simonl, monicat 99783 Separated out so department can keep their own address books, local to the group.
Inbound Fax - Receiving Sub Inbound/Outbound2 Placeholder for inbound route only; for receiving faxes destined for Receiving department copier (reccopier) 604-276-5250 faxreceiving@unipharm.com None 99645
Inbound Fax - Returns Sub Inbound/Outbound2 Placeholder for inbound route only; for receiving faxes destined for Returns department copier (rtncopier) 604-276-5283 faxreturns@unipharm.com None 99643
Inbound Fax - Elaho Sub Inbound/Outbound2 Placeholder for inbound route only; for receiving faxes destined for 2nd-floor copier (elaho) 604-270-9728 faxelaho@unipharm.com None 99639
Inbound Fax - Stein Sub Inbound/Outbound2 Placeholder for inbound route only; for receiving faxes destined for 1st-floor copier (stein) 604-270-8537 faxstein@unipharm.com None 99647
Angela Chan Sub Inbound/Outbound Inbound and outbound; dedicated for use by Angela only 604-270-2884 angelac@unipharm.com None 99571 Inbound faxes will print to angelacprinter
Ron Gracan Sub Inbound/Outbound Inbound and outbound; dedicated for use by Ron only 604-276-5255 rong@unipharm.com None 99579 Inbound faxes will print to rongprinter
1 SRFax requires aliases to have a username > 4 characters, yet the email address cannot be used (reserved for user accounts). So the user names for tomc and timm are 'Tom Chotwanwirach' and 'Tim Matiachuk' respectively. Will they notice? Lol c'mon now, it's fax.

2 Note that while these accounts are of the Inbound/Outbound type (billing-wise), the point is that they are being used for inbound purposes only.

Backups

Veeam

As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.

Hardware

On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:

Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

Veeam Software Installation

In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

Veeam Licensing

We will probably get perpetual licensing with renewable annual support

What Is Veeam Backing Up?

Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.

Backup Strategy And Scheduled Jobs

Placeholder something something something.

How Veeam Uses Tape Hardware

Placeholder something something something.

Veeam Technical Support

Put the terms of the support contract in here plus the phone number and expiry dates.

Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here

2018

  1. Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
  2. Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
  3. Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
  4. VMHost01DataStore01 created as the first datastore
  5. The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
  6. VCSA password is NewVisionIT@2051
  7. vcsa.unipharm.local is at 172.30.18.23
  8. VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
  9. VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
  10. Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
  11. Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
  12. vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
  13. 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
  14. Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
  15. Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
  16. Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
  17. Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
  18. Configured VUM to scan for updates each Monday of every week and email DarrenF
  19. Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
  20. P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
  21. Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
  22. Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
  23. P2Ved the Lucy physical server onto VMHost01 April 7, 2018
  24. P2Ved the Mail physical server onto VMHost01 April 15, 2018
  25. Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
  26. Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
  27. Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
  28. Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
  29. P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
  30. Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
  31. Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
  32. Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
  33. P2Ved the Smithers physical server onto VMHost01 May 19, 2018
  34. Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
  35. VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
  36. VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
  37. New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
  38. New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
  39. Thermoprofile virtual machine deleted from the datastore on June 13, 2018
  40. XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
  41. Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
  42. Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
  43. Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
  44. Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
  45. Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
  46. Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
  47. Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
  48. Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
  49. Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
  50. Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
  51. vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
  52. Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
  53. Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
  54. Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
  55. 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
  56. Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018

2019

  1. DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
  2. DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
  3. DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
  4. DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
  5. DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
  6. DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
  7. DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
  8. DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
Retrieved from "https://owl.unipharm.com/mediawiki/index.php?title=Information_Systems:VMWare_Production_Infrastructure&oldid=12539"