uniWIKI - User contributions [en]

Information Systems:Physical Security Door Codes

2019-07-29T19:55:18Z

Darrenf: Created page with "The number code to unlock the network closet next to Eliza's office and Monica's cube is: * Press 5 and release, then press 1 and release, then press 3 and 4 together and rele..."

The number code to unlock the network closet next to Eliza's office and Monica's cube is:
* Press 5 and release, then press 1 and release, then press 3 and 4 together and release, then press 2 and release

The number code to open the mezzanine door on the second floor office area is:
* Press 3 and 4 together and release, then press 5 and release, press 2 and release

The number code to open the Server Room door on the second floor mezzanine is:
* Press 5 and 1 together and release, press 4 and release, press 3 and release, press 2 and release

The number code to open the second floor storage room ( playroom ) AND the IT Cage is:
* Press 2 and 3 together and release, then press 1 and release, then press 5 and release

If you cannot open a door then call GeraldP, GordonM or JohnT as they all have keys to open the above doors.

Information Systems:Zabbix NMS

2019-07-25T19:06:14Z

Darrenf:

Zabbix is an open source software package installed on a Linux operating system or used as a pre-built virtual machine appliance. The software's purpose is to monitor and display primarily SNMP statistics from network switches or any network attached hardware that speaks SNMP. At uniPHARM, we are using Zabbix to gain a lot of visibility on to what is happening on our network switch infrastructure. Zabbix can query each network port on each network switch and read all of the available statistics such as packets in/out or ping time or bandwidth usage and a hundred other metrics. Zabbix can display all those ports, functions and numbers as graphs so that humans can see if there is a bottleneck or some other problem. Zabbix does have some logic built in that makes choices on what it thinks is an existing problem. For example, if a port on switch has a large number of errors or a port's bandwidth is pegged at maximum or even if a port is unplugged, it will generate an alert on the Zabbix dashboard and wait for somebody to act on it. Unfortunately, Zabbix is not able to make configuration changes on the switches directly so keep in mind that Zabbix is a "network monitoring server" and not a "network management server".

* http://zabbix.unipharm.local/zabbix
* Username is Admin
* Password is zabbix
* As of July 19 2019, AD logins are working but the above credentials are superior than an AD login. The Linux appliance username is "appliance".

The design of the Zabbix web interface does take a little getting used to. When not making any configuration changes to how Zabbix queries network attached hardware, stick to the "Monitoring" tab in the top left corner of the site. Within Monitoring you will be able to see one or more dashboards and a dedicated page for any listed Problems. You can also go to the Graphs tab and look and all the possible graphs by changing the group, host and graph drop down boxes. The Graphs tab is really the prime rib of the Zabbix application because the stat data can be presented in graphs that can go back as far as 2 years or 2 hours so trend lines can be visualsed easily.

The "Screens" tab is also extremely useful because it contains pre-built ... screens that show groups of graphs that are relevant together instead of individually. For example, one of the screens shows network traffic in/out plus errors on all of the wireless access points at uniPHARM. This shows which AP's have the most activity and shows what time of day that activity happens. The "Maps" tab shows a Visio style graphical layout of the network. Obviously its not as good as Visio but for an OSS web based mapper, its pretty good. Try and keep these diagrams current and up to date because they will be helpful should I.S. staff get hit by a bus at any time.

The object organization in Zabbix is also a little tricky to understand. There is a hierarchy which starts with "Host Groups" in the "Configuration" tab. A Host Group can contain one or more hosts plus one or more Templates. A template in Zabbix is ... a way of interpreting SNMP data. A template that correctly displays SNMP data from a Cisco switch is going to be different than a template that reads Juniper switch SNMP data. Luckily, most of the major hardware brands are included in Zabbix and there is also a bunch of generics that can also be used. Once you create a Host Group and assigned a Template you can create a Host. This is the part where Zabbix fails to be convenient. It's automatic network discovery function doesn't work and you have to manually create each network object. Yes, this is very time consuming and not convenient. In the Host creation step you can either use SNMP or the Zabbix agent to query stats. The agent is a small program that can be installed on a physical server or a VM but SNMP is best used for anything that's hardware. Zabbix will start to query the newly created host right away and depending on what it is and what template was applied, there may be many items, graphs, apps and triggers created that all work together to present as much information as possible.

Zabbix has been configured to send out email alerts when certain conditions are present. For example, if a switch or access point is not pingable, then Zabbix will create a problem ticket and send an email. The built in templates also allow much more fine grained alerts but I have tried to group things together logically. The alert configuration is in the Configuration/Actions tab. Here alerts can be turned off or on or edited. Its important to understand that if for example, a switch or server loses power and then comes back up, the problem ticket that Zabbix generates will be closed by the automated resolution ticket because the state of the hardware returned to baseline normal. If Zabbix is alerting on something that is a false positive like a wonky fan tachometer or a bad temperature sensor, then what needs to happen is the "Trigger Item" in the Host configuration needs to be disabled. That will mute an alert if it is included in an alert package.

As mentioned above, network discovery in Zabbix, at least for switches and printers doesn't work in an obvious way - it might but I couldn't figure it out. Discovery of the VMware infrastructure does indeed work and infact it works a little too well. The Host groups get a little untidy with how Zabbix displays virtual machines because it is showing by cluster and then by host so individual guests show up twice. Anywho, as of July 2019, the config of how Zabbix displays and tracks statistic info from the VMHosts and from the guests is correct. There are some yummy graphs in the Screens section and those screen should show anyone all the info they would ever need to see what's happening on the infrastructure.

Updating Zabbix. I don't know how to do this, so either leave the Linux appliance as is, which is safe, or do some command line advanced magic to do in place upgrades of the Zabbix ... files. Oh and there is some sort of sharing page on the public Zabbix site where new or customized templates are available, that may be helpful in the future if we are using hardware that isn't included by default. A lot of hours were put into getting Zabbix usable, please take care of it, at the very least take a snapshot before doing anything crazy so that you can revert back.

=Things Still To Do=
* Zabbix does apparently speak WMI to Windows machines but I could not figure out how to do this because the documentation is not great. It would be useful for Zabbix to get WMI info.
* Better map icons. The default icons that are available in Zabbix are very limited and don't make creating maps as easy as it could. There is a way to import more icons/shapes but I wasn't able to get that far
* External monitoring - this one appears to be super complicated but there may be potential for Zabbix to replace the 24x7 Zoho service we use. Of course the downside of that is that because Zabbix is a VM it is internal and dependent on everything else working to be able to properly monitor external sites. The extended thought is that Zabbix as a "thing" won't be able to alert on anything if either the VM hangs or the host it is on hangs or is otherwise down itself. Funny how we didn't build in that redundancy to the VMware environment.

[[Category: I.T. Projects and Ideas]]
[[Category: Power, Alarms, and Monitoring]]

Information Systems:Zabbix NMS

2019-07-25T18:36:19Z

Darrenf:

Zabbix is an open source software package installed on a Linux operating system or used as a pre-built virtual machine appliance. The software's purpose is to monitor and display primarily SNMP statistics from network switches or any network attached hardware that speaks SNMP. At uniPHARM, we are using Zabbix to gain a lot of visibility on to what is happening on our network switch infrastructure. Zabbix can query each network port on each network switch and read all of the available statistics such as packets in/out or ping time or bandwidth usage and a hundred other metrics. Zabbix can display all those ports, functions and numbers as graphs so that humans can see if there is a bottleneck or some other problem. Zabbix does have some logic built in that makes choices on what it thinks is an existing problem. For example, if a port on switch has a large number of errors or a port's bandwidth is pegged at maximum or even if a port is unplugged, it will generate an alert on the Zabbix dashboard and wait for somebody to act on it. Unfortunately, Zabbix is not able to make configuration changes on the switches directly so keep in mind that Zabbix is a "network monitoring server" and not a "network management server".

* http://zabbix.unipharm.local/zabbix
* Username is Admin
* Password is zabbix
* As of July 19 2019, AD logins are working but the above credentials are superior than an AD login. The Linux appliance username is "appliance".

The design of the Zabbix web interface does take a little getting used to. When not making any configuration changes to how Zabbix queries network attached hardware, stick to the "Monitoring" tab in the top left corner of the site. Within Monitoring you will be able to see one or more dashboards and a dedicated page for any listed Problems. You can also go to the Graphs tab and look and all the possible graphs by changing the group, host and graph drop down boxes. The Graphs tab is really the prime rib of the Zabbix application because the stat data can be presented in graphs that can go back as far as 2 years or 2 hours so trend lines can be visualsed easily.

The "Screens" tab is also extremely useful because it contains pre-built ... screens that show groups of graphs that are relevant together instead of individually. For example, one of the screens shows network traffic in/out plus errors on all of the wireless access points at uniPHARM. This shows which AP's have the most activity and shows what time of day that activity happens. The "Maps" tab shows a Visio style graphical layout of the network. Obviously its not as good as Visio but for an OSS web based mapper, its pretty good. Try and keep these diagrams current and up to date because they will be helpful should I.S. staff get hit by a bus at any time.

The object organization in Zabbix is also a little tricky to understand. There is a hierarchy which starts with "Host Groups" in the "Configuration" tab. A Host Group can contain one or more hosts plus one or more Templates. A template in Zabbix is ... a way of interpreting SNMP data. A template that correctly displays SNMP data from a Cisco switch is going to be different than a template that reads Juniper switch SNMP data. Luckily, most of the major hardware brands are included in Zabbix and there is also a bunch of generics that can also be used. Once you create a Host Group and assigned a Template you can create a Host. This is the part where Zabbix fails to be convenient. It's automatic network discovery function doesn't work and you have to manually create each network object. Yes, this is very time consuming and not convenient. In the Host creation step you can either use SNMP or the Zabbix agent to query stats. The agent is a small program that can be installed on a physical server or a VM but SNMP is best used for anything that's hardware. Zabbix will start to query the newly created host right away and depending on what it is and what template was applied, there may be many items, graphs, apps and triggers created that all work together to present as much information as possible.

Zabbix has been configured to send out email alerts when certain conditions are present. For example, if a switch or access point is not pingable, then Zabbix will create a problem ticket and send an email. The built in templates also allow much more fine grained alerts but I have tried to group things together logically. The alert configuration is in the Configuration/Actions tab. Here alerts can be turned off or on or edited. Its important to understand that if for example, a switch or server loses power and then comes back up, the problem ticket that Zabbix generates will be closed by the automated resolution ticket because the state of the hardware returned to baseline normal. If Zabbix is alerting on something that is a false positive like a wonky fan tachometer or a bad temperature sensor, then what needs to happen is the "Trigger Item" in the Host configuration needs to be disabled. That will mute an alert if it is included in an alert package.

As mentioned above, network discovery in Zabbix, at least for switches and printers doesn't work in an obvious way - it might but I couldn't figure it out. Discovery of the VMware infrastructure does indeed work and infact it works a little too well. The Host groups get a little untidy with how Zabbix displays virtual machines because it is showing by cluster and then by host so individual guests show up twice. Anywho, as of July 2019, the config of how Zabbix displays and tracks statistic info from the VMHosts and from the guests is correct. There are some yummy graphs in the Screens section and those screen should show anyone all the info they would ever need to see what's happening on the infrastructure.

Updating Zabbix. I don't know how to do this, so either leave the Linux appliance as is, which is safe, or do some command line advanced magic to do in place upgrades of the Zabbix ... files. Oh and there is some sort of sharing page on the public Zabbix site where new or customized templates are available, that may be helpful in the future if we are using hardware that isn't included by default. A lot of hours were put into getting Zabbix usable, please take care of it, at the very least take a snapshot before doing anything crazy so that you can revert back.

[[Category: I.T. Projects and Ideas]]
[[Category: Power, Alarms, and Monitoring]]

Information Systems:M-Daemon Mail Server

2019-07-24T16:59:03Z

Darrenf: /* Overview */

=Overview=
MDaemon is a full featured email server that can be considered an alternative to Exchange or Domino or even O365. MDaemon is able to serve as an SMTP, IMAP, POP3 host and can handle many hundreds or thousands of email accounts. The software has a fully featured webmail interface called WorldClient and a web based administration page that mimics most of the configuration options that exist in the GUI. At uniPHARM, the practice is to use the MDaemon GUI to do administrative tasks such as creating new email accounts or watching server activity in real time. MDaemon also acts as a smart host for IBM Domino standing between Domino and the public internet. MDaemon is positioned in this manner because configuring its security settings are much easier compared to Domino. Getting logs from MDaemon is also considerably easier compared to Domino and how the GUI displays configuration options just appears to be more logical and easier to use compared to Domino, plus its safer to expose SMTP, IMAP, POP3 ports from the public internet to MDaemon than allowing all that dirty public traffic onto an interface on the Power8.

Be aware that although the number of email orders flowing from pharmacies through Barracuda to MDaemon has been reduced, there are still some shareholders that send email orders and they do get received by MDaemon and then get forwarded to the Power8 for processing by IMS IMP and IOP. If MDaemon is offline then email flow will stop for everything inbound and outbound. The other big concept to understand is that while MDaemon can stand on its own and do everything related to email, we don't use any of its anti-spam features because Barracuda is responsible for that. MDaemon does scan each inbound and outbound email with its internal anti-virus scanner and can quarantine messages that contain a virus, but its not doing any filtering based on the spaminess of any message. uniPHARM renews the yearly support contract for MDaemon with C&C Software in Ontario each August so that we can call them for support (rare) and be entitled to version upgrades for MDaemon and the Security Plus addon which does the AV scanning. Upgrading MDaemon and Security Plus is always fairly straight forward - stop the services and run each installer. The maker of MDaemon is a company called Alt-N and they put in a lot of effort to make the upgrade installers work safely and consistently. Alt-N is also very aggressive at creating updated versions with bug fixes for the existing code.

If our install of MDaemon does need to be moved to a new VM, the daily configuration backup is going to be absolutely critical because that config backup holds all the settings for MDaemon. According to Alt-N, in order to move the install you first stop all services do an in place upgrade to the latest version, copy the C:\MDaemon folder to the new VM, run the latest installer version again on the new VM and reboot. The new VM would ideally have the same IP and host name and DNS settings as the old VM (which would be off) and that should be enough to get MDaemon up and responding on a new VM. Also, if a move is needed, its necessary to understand that there are shareholder pharmacies that do use our MDaemon server for all of their email needs. Some pharmacies are using O365 or Gsuite but some are dependent on MDaemon and any configuration changes need to have that in mind.

The other important thing to note about the Mail VM is that even though the Administrator username and the password are the same as the domain admin credentials, this VM is not a domain member server. It is a standalone server not connected to Active Directory because that would require too many holes punched into the firewall DMZ, which is where this VM lives on the network. We were looking at a project earlier in 2019 to move the Mail VM out of the DMZ and to close down the DMZ because it's just not needed anymore. That all got shelved for higher priority stuff so just keep in mind that Mail is not connected to AD and doesn't get any GPOs but the local administrator password happens to be the same as the domain admin password.

=World Client=
Web Browser Email For Shareholders

MDaemon has a built in web server that is installed on the mail server. The web server is secured with an SSL certificate and the URL is

https://mail.unipharm.com

This website shows up as “World Client”. Think of it as iNotes for MDaemon. Any email account in MDaemon is accessible using this website. The username is the full email address and the password is the password for that email account.

For example

Username test@unipharm.com
Password bed

Only a few shareholders use this service, the majority still use various flavours of Outlook or some other email program. Internet Explorer version 10 and 11 and recent versions of Firefox are required for the World Client website to work. IE 6,7,8,9 will not work.
=Problems=
Shareholder pharmacies may call in with an email related question that turns out to be an unknown password. MDaemon does not display email account passwords unless they are exported into a text file along with all of the rest of the account parameters. Below is a step by step procedure to view what a password is for email accounts in MDaemon. It seems complicated but its not.
* Remote desktop into the Mail virtual machine. The domain administrator username and password are in the spreadsheet on the SuperServer
* Open the red circle icon for the MDaemon GUI
* Click on the Accounts menu and then Exporting and then Export accounts into a comma delimited file
* Click Yes
* Logon is uwdpostmaster
* Password is NewVisionIT
* Wait 5 seconds ... yes I'm serious
* Minimize the MDaemon GUI and you will see a text file on the Desktop called Accounts Text File - open that
* Ctrl-F to search for the email account
* Then look at the sixth comma separated field - that's the password. If the field is blank then the password is blank.
* When you close the text file, the file is deleted by MDaemon however the shortcut to the file remains on the Desktop. Yes I know its odd behavior.

==Locked Accounts==
MDaemon may automatically accounts based on certain activity patterns. One such rule is the attempt SMTP sending of 250 outbound messages in 30 minutes. This indicates a hijacked account, and requires that the password be changed for the account. Following, this any active SMTP sessions (likely many if this is a botnet hijack) need to be terminated, and this is best done by restarting the SMTP server component of MDaemon.

[[Category: Email]]
[[Category: Servers - Software]]
[[Category: System Administration]]

Information Systems:LAN infrastructure at uniPHARM

2019-07-18T23:02:32Z

Darrenf:

==Overview==
This is the main page for the LAN (wired network configuration) at uniPHARM.

==Switches==
This table outlines the physical network switches that make up the network.

{| class="wikitable sortable"
|-
! Hostname !! IP address || Switch model !! Location !! Username !! Password !! Notes
|-
| '''coresw.unipharm.local''' || 172.30.16.2 || 3Com 3870 || Server room || admin || visionit || '''Core switch.''' Stack of 4.
|-
| atpsw.unipharm.local || 172.30.16.3 || 3Com 4200 || Server room || admin || visionit || Feeds data to POE injectors for Mirador temp. sensors
|-
| accountingsw.unipharm.local || 172.30.16.11 || Cisco SG300-28P || Accounting Wallmount Network Rack || adminit || NewVisionIT2051 || POE, ~175W power budget
|-
| buyersw.unipharm.local || 172.30.16.5 || 3Com 4200G? || Buying department closet || admin || visionbs || -
|-
| itsw.unipharm.local || 172.30.16.6 || 3Com 4200G || IT department ceiling tile. Yup. || admin || visionis || -
|-
| dumpsw.unipharm.local || 172.30.16.7 || 3Com 4200G? || Electrical Room? || admin || visionds || -
|-
| pickingsw.unipharm.local || 172.30.16.8 || 3Com 3870? || Rx picking zone network rack || admin || visionps || -
|-
| receivingsw.unipharm.local || 172.30.16.10 || 3Com 4200G? || Outside Gerald's office, ceiling tile || admin || visionrs || -
|-
| wallsw.unipharm.local || 172.30.16.7 || 3Com 4200G? || Candy Mezzanine || admin || visionns || Functions merely to bridge the physical wiring between Receiving area and core network
|-
| jeremymsw || 172.30.16.16 || TP-Link SG105E || Jeremy's office || admin || visionit || Used for testing Yealink, web-managed smart switch
|-
| norwinusw || 172.30.16.15 || Netgear GS105PE || Norwin's office || - || visionit || Used to access multiple VLANs, network administration, and testing. This switch has PoE passthrough.
|}

* External link to backdoor access and hidden menus for the 3com 3870 switches http://etherhack.wikia.com/wiki/3Com_3870
* There is a spare, ready to use 3com 3870 24 port switch located at the very bottom of the networking rack in the server room. The switch has been reset to factory defaults and the username is admin and the password is blank. The switch is ready to be put into the core stack if needed.

==VLANs==
We started using VLANs in 2016, for the same reason why anyone would use VLANs - to manage/separate multiple networks using one physical switch infrastructure. The VLAN implementation being used is the standard 802.1q. The following table outlines the VLAN infrastructure.

{| class="wikitable sortable"
|-
! VLAN ID !! Name !! L3 Network / IP Range !! Description !! Notes
|-
| '''1''' || Main LAN || 172.30.16.0/21 (172.30.16.1 - 172.30.23.255) || Default VLAN, untagged across the network for simplicity (so that every device doesn't have to be configured to talk this VLAN, or even be VLAN-aware at all). || Routed out Telus Fibre
|-
| '''2''' || Staff and guest WiFi || 192.168.0.1/23 (192.168.0.1 - 192.168.1.254) || Colloquially known as the "Guest VLAN", but staff wifi devices are not technically guest devices. Currently, shipping computers are on this network, but they should be moved to VLAN 7 when it is created. || Routed out Shaw Business Internet
|-
| '''3''' || vMotion LAN || ?? || Small network to isolate vMotion traffic between the 3 virtual hosts. ||
|-
| '''4''' || Shaw WAN || Network of the Shaw static WAN IP (/22?) || Network to isolate WAN traffic between Shaw modem and Sophos WAN interface from the rest of the network. Lots of ARP going on here! ||
|-
| '''5''' || Telus WAN || Network of our Telus WAN IPs (/27) || Network to isolate WAN traffic between Telus modem and Sophos WAN interface from the rest of the network. ||
|-
| '''6''' || VoIP LAN || 192.168.2.0/24 || Contains PBX and IP phones || Routed out dedicated Shaw SIP internet connection. ||
|-
| '''7''' || Alt LAN || 192.168.3.0/24 || Network containing business devices that we do not want in our main LAN e.g. shipping terminals, digital signage player || Routed out Shaw Business Internet, uses Telus Fibre as a backup.
|}
=Helpful Stuff=
The command to view what MAC address(s) is in use on a port for the Cisco SG350 switch is as follows. You will need to login to the switch via SSH in order to do this:
* show mac address-table interface gi1 (or ge1 sometimes too)
The port name is visible from the switches web interface and goes from gi1 to gi28 or ge1 to ge28. Then cross checking the MAC address in Spiceworks for example will allow you to see who is plugged into what port - however all the Cisco switches should already have human readable labels attached to all in-use ports as of July 2019.
[[Category: Networking]]

Information Systems:M-Daemon Mail Server

2019-07-18T21:37:32Z

Darrenf: /* Problems */

=Overview=
MDaemon is a full featured email server that can be considered an alternative to Exchange or Domino or even O365. MDaemon is able to serve as an SMTP, IMAP, POP3 host and can handle many hundreds or thousands of email accounts. The software has a fully featured webmail interface called WorldClient and a web based administration page that mimics most of the configuration options that exist in the GUI. At uniPHARM, the practice is to use the MDaemon GUI to do administrative tasks such as creating new email accounts or watching server activity in real time. MDaemon also acts as a smart host for IBM Domino standing between Domino and the public internet. MDaemon is positioned in this manner because configuring its security settings are much easier compared to Domino. Getting logs from MDaemon is also considerably easier compared to Domino and how the GUI displays configuration options just appears to be more logical and easier to use compared to Domino, plus its safer to expose SMTP, IMAP, POP3 ports from the public internet to MDaemon than allowing all that dirty public traffic onto an interface on the Power8.

Be aware that although the number of email orders flowing from pharmacies through Barracuda to MDaemon has been reduced, there are still some shareholders that send email orders and they do get received by MDaemon and then get forwarded to the Power8 for processing by IMS IMP and IOP. If MDaemon is offline then email flow will stop for everything inbound and outbound. The other big concept to understand is that while MDaemon can stand on its own and do everything related to email, we don't use any of its anti-spam features because Barracuda is responsible for that. MDaemon does scan each inbound and outbound email with its internal anti-virus scanner and can quarantine messages that contain a virus, but its not doing any filtering based on the spaminess of any message. uniPHARM renews the yearly support contract for MDaemon with C&C Software in Ontario each August so that we can call them for support (rare) and be entitled to version upgrades for MDaemon and the Security Plus addon which does the AV scanning. Upgrading MDaemon and Security Plus is always fairly straight forward - stop the services and run each installer. The maker of MDaemon is a company called Alt-N and they put in a lot of effort to make the upgrade installers work safely and consistently. Alt-N is also very aggressive at creating updated versions with bug fixes for the existing code.

If our install of MDaemon does need to be moved to a new VM, the daily configuration backup is going to be absolutely critical because that config backup holds all the settings for MDaemon. According to Alt-N, in order to move the install you first stop all services do an in place upgrade to the latest version, copy the C:\MDaemon folder to the new VM, run the latest installer version again on the new VM and reboot. The new VM would ideally have the same IP and host name and DNS settings as the old VM (which would be off) and that should be enough to get MDaemon up and responding on a new VM. Also, if a move is needed, its necessary to understand that there are shareholder pharmacies that do use our MDaemon server for all of their email needs. Some pharmacies are using O365 or Gsuite but some are dependent on MDaemon and any configuration changes need to have that in mind.

=World Client=
Web Browser Email For Shareholders

MDaemon has a built in web server that is installed on the mail server. The web server is secured with an SSL certificate and the URL is

https://mail.unipharm.com

This website shows up as “World Client”. Think of it as iNotes for MDaemon. Any email account in MDaemon is accessible using this website. The username is the full email address and the password is the password for that email account.

For example

Username test@unipharm.com
Password bed

Only a few shareholders use this service, the majority still use various flavours of Outlook or some other email program. Internet Explorer version 10 and 11 and recent versions of Firefox are required for the World Client website to work. IE 6,7,8,9 will not work.
=Problems=
Shareholder pharmacies may call in with an email related question that turns out to be an unknown password. MDaemon does not display email account passwords unless they are exported into a text file along with all of the rest of the account parameters. Below is a step by step procedure to view what a password is for email accounts in MDaemon. It seems complicated but its not.
* Remote desktop into the Mail virtual machine. The domain administrator username and password are in the spreadsheet on the SuperServer
* Open the red circle icon for the MDaemon GUI
* Click on the Accounts menu and then Exporting and then Export accounts into a comma delimited file
* Click Yes
* Logon is uwdpostmaster
* Password is NewVisionIT
* Wait 5 seconds ... yes I'm serious
* Minimize the MDaemon GUI and you will see a text file on the Desktop called Accounts Text File - open that
* Ctrl-F to search for the email account
* Then look at the sixth comma separated field - that's the password. If the field is blank then the password is blank.
* When you close the text file, the file is deleted by MDaemon however the shortcut to the file remains on the Desktop. Yes I know its odd behavior.

==Locked Accounts==
MDaemon may automatically accounts based on certain activity patterns. One such rule is the attempt SMTP sending of 250 outbound messages in 30 minutes. This indicates a hijacked account, and requires that the password be changed for the account. Following, this any active SMTP sessions (likely many if this is a botnet hijack) need to be terminated, and this is best done by restarting the SMTP server component of MDaemon.

[[Category: Email]]
[[Category: Servers - Software]]
[[Category: System Administration]]

Information Systems:Zabbix NMS

2019-07-17T18:44:03Z

Darrenf:

Information Systems:Zabbix NMS

2019-07-17T18:43:13Z

Darrenf:

Information Systems:Zabbix NMS

2019-07-17T18:42:03Z

Darrenf: Created page with "Zabbix is an open source software package installed on a Linux operating system or used as a pre-built virtual machine appliance. The software's purpose is to monitor and dis..."

Information Systems:VMWare Production Infrastructure

2019-07-17T18:07:00Z

Darrenf: /* vCenter Management */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui <------ The HTML5 site is waaaaaaaaay better than the Flash site AND it has a dark mode them which is bananas.
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

==Zabbix NMS VM Appliance==
This is a Linux based virtual appliance that is used for network monitoring, graphing and alerting. The VM appliance is an LTS version of Ubuntu with the OSS package called Zabbix baked in. The open-vm-tools package is present so vCenter can control this VM. While this VM is not business critical, it took a lot of effort to setup and configure so try not to let it get wiped out with no backups.
* The virtual machine is configured with 2 vCPU, 4GB of memory and 40GB of thin provisioned disk space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.13
* As of July 2019, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
The following chart describes where a virtual machine should be located so as to balance the processing, memory and storage loads evenly among the 3 hosts in the cluster. VM's can be moved around or shuffled from host to host when doing maintenance but this is the preferred layout of which hosts house which VM's. Changes to this chart will be documented in the change log at the bottom of this page.
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03
|-
| XClarity || SuperServer || 3CX Appliance
|-
| Lucy || UWDDC2 || TestServer1
|-
| Mail || VMware vCenter Server Appliance || UbuntuDevVM
|-
| Smithers || XTGUI || Windows7 Ent For Adobe LifeCycle
|-
| UWDDC1 || Zabbix NMS VM Appliance || WindowsXP Pro HP JetDirect
|-
| WDS || || Windows10 Ent Eval
|-
| || || Windows10 Pro Eval
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
# DarrenF created a new VM for the Zabbix NMS VM Appliance using an ISO located in the Content Library - July 10, 2019
# DarrenF enabled SNMP on all three hosts and the VCSA so that Zabbix could monitor via SNMP - July 12, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-07-17T18:04:59Z

Darrenf: /* Zabbix NMS VM Appliance */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

==Zabbix NMS VM Appliance==
This is a Linux based virtual appliance that is used for network monitoring, graphing and alerting. The VM appliance is an LTS version of Ubuntu with the OSS package called Zabbix baked in. The open-vm-tools package is present so vCenter can control this VM. While this VM is not business critical, it took a lot of effort to setup and configure so try not to let it get wiped out with no backups.
* The virtual machine is configured with 2 vCPU, 4GB of memory and 40GB of thin provisioned disk space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.13
* As of July 2019, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
The following chart describes where a virtual machine should be located so as to balance the processing, memory and storage loads evenly among the 3 hosts in the cluster. VM's can be moved around or shuffled from host to host when doing maintenance but this is the preferred layout of which hosts house which VM's. Changes to this chart will be documented in the change log at the bottom of this page.
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03
|-
| XClarity || SuperServer || 3CX Appliance
|-
| Lucy || UWDDC2 || TestServer1
|-
| Mail || VMware vCenter Server Appliance || UbuntuDevVM
|-
| Smithers || XTGUI || Windows7 Ent For Adobe LifeCycle
|-
| UWDDC1 || Zabbix NMS VM Appliance || WindowsXP Pro HP JetDirect
|-
| WDS || || Windows10 Ent Eval
|-
| || || Windows10 Pro Eval
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
# DarrenF created a new VM for the Zabbix NMS VM Appliance using an ISO located in the Content Library - July 10, 2019
# DarrenF enabled SNMP on all three hosts and the VCSA so that Zabbix could monitor via SNMP - July 12, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-07-17T18:04:17Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

==Zabbix NMS VM Appliance==
This is a Linux based virtual appliance that is used for network monitoring, graphing and alerting. The VM appliance is an LTS version of Ubuntu with the OSS package called Zabbix baked in. The open-vm-tools package is present so vCenter can control this VM. While this VM is not business critical, it took a lot of effort to setup and configure so try not to let it get wiped out with no backups.
* The virtual machine is configured with 2 vCPU, 4GB of memory and 40GB of thin provisioned disk space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.13
* As of July 2019, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost01

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
The following chart describes where a virtual machine should be located so as to balance the processing, memory and storage loads evenly among the 3 hosts in the cluster. VM's can be moved around or shuffled from host to host when doing maintenance but this is the preferred layout of which hosts house which VM's. Changes to this chart will be documented in the change log at the bottom of this page.
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03
|-
| XClarity || SuperServer || 3CX Appliance
|-
| Lucy || UWDDC2 || TestServer1
|-
| Mail || VMware vCenter Server Appliance || UbuntuDevVM
|-
| Smithers || XTGUI || Windows7 Ent For Adobe LifeCycle
|-
| UWDDC1 || Zabbix NMS VM Appliance || WindowsXP Pro HP JetDirect
|-
| WDS || || Windows10 Ent Eval
|-
| || || Windows10 Pro Eval
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
# DarrenF created a new VM for the Zabbix NMS VM Appliance using an ISO located in the Content Library - July 10, 2019
# DarrenF enabled SNMP on all three hosts and the VCSA so that Zabbix could monitor via SNMP - July 12, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-07-17T17:59:27Z

Darrenf:

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

==Zabbix NMS VM Appliance==
This is a Linux based virtual appliance that is used for network monitoring, graphing and alerting. The VM appliance is an LTS version of Ubuntu with the OSS package called Zabbix baked in. The open-vm-tools package is present so vCenter can control this VM. While this VM is not business critical, it took a lot of effort to setup and configure so try not to let it get wiped out with no backups.
* The virtual machine is configured with 2 vCPU, 4GB of memory and 40GB of thin provisioned disk space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.13
* As of July 2019, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost01

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
The following chart describes where a virtual machine should be located so as to balance the processing, memory and storage loads evenly among the 3 hosts in the cluster. VM's can be moved around or shuffled from host to host when doing maintenance but this is the preferred layout of which hosts house which VM's. Changes to this chart will be documented in the change log at the bottom of this page.
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03
|-
| XClarity || SuperServer || 3CX Appliance
|-
| Lucy || UWDDC2 || TestServer1
|-
| Mail || VMware vCenter Server Appliance || UbuntuDevVM
|-
| Smithers || XTGUI || Windows7 Ent For Adobe LifeCycle
|-
| UWDDC1 || || WindowsXP Pro HP JetDirect
|-
| WDS || || Windows10 Ent Eval
|-
| || || Windows10 Pro Eval
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
# DarrenF created a new VM for the Zabbix NMS VM Appliance using an ISO located in the Content Library - July 10, 2019
# DarrenF enabled SNMP on all three hosts and the VCSA so that Zabbix could monitor via SNMP - July 12, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-07-12T20:39:59Z

Darrenf: /* 2019 */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
The following chart describes where a virtual machine should be located so as to balance the processing, memory and storage loads evenly among the 3 hosts in the cluster. VM's can be moved around or shuffled from host to host when doing maintenance but this is the preferred layout of which hosts house which VM's. Changes to this chart will be documented in the change log at the bottom of this page.
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03
|-
| XClarity || SuperServer || 3CX Appliance
|-
| Lucy || UWDDC2 || TestServer1
|-
| Mail || VMware vCenter Server Appliance || UbuntuDevVM
|-
| Smithers || XTGUI || Windows7 Ent For Adobe LifeCycle
|-
| UWDDC1 || || WindowsXP Pro HP JetDirect
|-
| WDS || || Windows10 Ent Eval
|-
| || || Windows10 Pro Eval
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
# DarrenF enabled SNMP on all three hosts and the VCSA so that Zabbix could monitor via SNMP - July 12, 2019.
[[Category: Virtualization]]

Information Systems:IBM Passport Advantage

2019-07-12T16:30:16Z

Darrenf: /* Contact information */

=Overview=

IBM Passport Advantage is an online portal for viewing software entitlements and downloading purchased software.

=Contact information=

*Site ID: 7404165
*Customer number: 090830

The customer number above is needed to get support from IBM on Domino and other IBM software that we have valid support contracts for.

[[Category: Pages with Contact Information]]

Information Systems:Xerox printers

2019-07-11T17:49:10Z

Darrenf: /* Xerox Fleet Management */

uniPHARM uses Xerox printers for office printing (as opposed to Lexmark printers used in the warehouse). These printers are under either lease or managed print service agreements:

{| class="wikitable sortable"
|-
! Printer hostname !! Model !! Location / purpose !! Contract type
|-
| reccopier || Xerox VersaLink B405DN || Receiving area office || Managed Print Services
|-
| rtncopier || Xerox VersaLink B405DN || Returns department || Managed Print Services
|-
| elaho || Xerox Versa C60 || 2nd-floor printing alcove || Lease
|-
| stein || Xerox AltaLink B8055 || 1st-floor [[:Category:uniPHARM Dictionary|dumps area]] || Lease
|}

==Support info==

* Contact number for leased printers (Xerox One Number): 1-800-275-9376
* Contact number for printers under Managed Print Services: 1-866-487-4239
* Contract end date: December 2023 (?)

==Old printers==
This section is for reference purposes. These printers are no longer being used.

There are 4 different Xerox copier models in use as of 2016. The WorkCentre 7775 is located in the printing alcove on the second floor. It uses colour and black powder toner and has an attached finisher that can fold paper. The colour touch screen is used to setup copy, scan and fax options. The 7775 has 2 fax lines connected so that it can fax inbound and outbound at the same time. The 7775 is leased and Xerox provides consumable toner and other supplies as part of the lease cost. Xerox also provides technical support and repairs as part of the lease. When requesting supplies, repairs or support you must provide the serial number which can be found on the copiers admin page.

:[http://elaho.unipharm.local/ Elaho Administration Web GUI]

*admin
*1111

Supplies Phone Number 1-800-275-9376
Support Phone Number 1-800-275-9376

The WorkCentre 5855 is located next the Supervisors Office in the shipping area of the DC. It has no colour capability but it does have the ability to fax inbound/outbound. The supplies the 5855 uses are different from the 7775 and there is a much smaller and less complicated touch screen. The 5855 is also on the same lease program as the 7775.

:[http://stein.unipharm.local/ Stein Administration Web GUI]

*admin
*1111

The WorkCentre 4265 copiers are located in the Receiving Office and the Returns Areas in the DC. They are smaller copiers with fewer paper trays. They both do have a single fax phone line attached and have very similar software capabilities compared to the 5855 and are on the same lease program as the 5855 and 7775.

:[http://reccopier.unipharm.local/ Reccopier Administration Web GUI]
:[http://rtncopier.unipharm.local/ Rtncopier Administration Web GUI]

*admin
*1111

The WorkCentre 6605 copiers are slightly smaller than the 4265s but otherwise perform and behave in the same way. The 6605s were free from Xerox and are NOT on the lease program and toner and supplies DO need to be purchased for the 2 that are used by the payroll administrator and general manager. Both of the 6605s are network and fax attached.

==Xerox Fleet Management==
This is a useful portal site that Xerox provides for leased or managed printers
* https://office.services.xerox.com/FMP/LoginPage.aspx
* Username is darrenf@unipharm.com
* Password is NewVisionIT@2019

==Xerox Device Agent Software===
There is an application from Xerox called the Xerox Device Agent, installed on the WDS virtual machine. This program sends the page meter reads for the copiers in Returns and Receiving to WestX for billing purposes. This application consumes a lot of CPU time. DarrenF tried for 4 months to get Xerox and its developers to figure out why a crappy little program uses so much CPU 24/7. Xerox wasn't willing to acknowledge a problem or allow a conversation to happen with their in-house developers - so the support ticket was closed as unresolved. The Xerox Device Agent software needs to continue to run and report billing data to WestX so if the CPU usage becomes a problem, try to get a new version of the app or open a new ticket with Xerox and prepare for pain.

==Xerox NeckBeard Support==
On June 13th, 2019, a Xerox technician was called onsite to fix the C60 that was not copying paper in the top feeder or from the glass. He determined that the OS was corrupt and did a factory reset of the machine which wiped out all of its configuration settings. He failed to notify anyone that he was doing a factory reset and he failed to create a backup clone file of the configuration to restore. This caused a great deal of inconvenience and extra work that was not necessary if a clone file was available. If this dumbfuckery ever happens again, there are saved clone files on the SuperServer in the following location : \\superserver.unipharm.local\Tech\Logs And Backups\Xerox Copier Clone Files\. In that folder are clone files for all the leased and MPS copiers as of June 2019. The 2 Xerox MFDs that are used by RonG and AngelaC cannot produce clone files. In case you haven't caught on, a clone file is a file that contains all the settings that are configured on a printer, with the purpose of "cloning" itself onto another printer of the same model or the exact same one after some asshat decides to do something idiotic.

[[Category: Printing]]
[[Category: Pages with Contact Information]]

Information Systems:Network Monitoring

2019-07-04T16:23:57Z

Darrenf:

==Software considerations==
* Paessler
* Wireshark/TAP interface
* Spiceworks

Spiceworks is installed on the WDS virtual machine running as a Windows service. The application is a swiss army knife for IT stuff. It can inventory a network and see and discover servers, desktops, laptops, printers etc. It can also do some rudimentary monitoring and alerting and can even keep track of vendor renewals and purchases. Spiceworks is a good and useful application, but because it does so much, it doesn't really do anything really well. If WMI is all gunked up on a workstation or server, it won't display any useful information. If a printer or copier has wacky SNMP, it will also not display anything important. The good stuff in the inventory is that (as long as the IT person remembers) SW can record the date of purchase, the dollar amount and the asset number of something. This is a great way to figure out how old tech assets are, because SW can output a depreciation report. Ultimately it depends on how accurate the data is input. For example, if a new network switch is purchased, part of the mental setup process is to allow SW to do an inventory scan so it can discover the new hardware. SW will fill in items such as serial number and MAC address because that's either in SNMP or WMI, it is up to the person who is setting up the new hardware to fill in the purchase date, dollar amount and asset number. Keeping accurate records pays off when the asset reaches its end of life and has to be recycled. Knowing the asset number linked to the serial number makes everybody's work easier.

SW does a mostly ok job at keeping track of things like server warranties and contract and support renewals. It tries to figure out what vendor a server is from (Lenovo, HPE, Dell e.g.) and tries to do a serial number lookup on the vendors warranty page and if SW gets a valid answer it will show the expiry date of any applicable warranties. Contracts and support renewals do have to be entered and UPDATED manually but again, if the input is accurate, so will the output. SW has great potential to really be a fantastic application but as of 2019, it is ad-driven and the company doing development is focusing on a cloud version instead of the on premise version. If anyone in the IT dept at UWD gets hit by a bus, Spiceworks will help their replacement get a good picture of the environment.

* http://wds.unipharm.local:81/pro_users/login
* darrenf@unipharm.com
* visionit

[[Category: I.T. Projects and Ideas]]

Information Systems:Printers And Picking Stations In The DC

2019-07-02T17:37:31Z

Darrenf:

=IF YOU CHANGE ANYTHING IN THE DC LIKE SWAPPING OUT PRINTERS OR PUTTING A BROKEN PRINTER IN THE CAGE - YOU HAVE TO RECORD THAT HERE=
* This page is current as of July 2, 2019

=Production Printer Fleet Totals=
* Lexmark MS415 invoice printer - 1
* Lexmark T650 invoice printers - 8
* Lexmark T640 invoice printers - 2
* Lexmark T630 invoice printers - 1
* Zebra S4M direct thermal label printers - 20
* Zebra 2746e label printers - 4 plus 1 using a ribbon for a total of 5
* Zebra small label printers - 1(GX420t) 1(LP2242) 1(TLP2742) 1(TLP2844) in Returns and Receiving

=Ready Spares Printer Fleet Totals=
* Lexmark MS521 new in box - 3
* Lexmark T650 spare invoice printers - 2
* Lexmark T640 spare invoice printers - 2
* Lexmark T630 spare invoice printers - 1
* Zebra S4M ready spare - 5
* Zebra 2746e ready spare - 2
* Broken printers of any type ready to be recycled - 2

=Production Picking Stations=
* OTC picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HABA picking stations has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* LargeDown picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Lower90s picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Pharmies picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HomeHealthCare picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* FridgeArea picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* NarcCage picking station has 1 Lexmark MS415 and 2 Zebra S4M all connected via parallel cable to an HP print box (Lexmark MS410 also network attached)
* Candy picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* SalesArea picking station has 1 Lexmark T650 and 2 Zebra 2746e all connected via parallel cable to an HP print box
* TobaccoArea picking station has 1 Lexmark T630 and 2 Zebra 2746e all connected via parallel cable to an HP print box (yes they still pick here)

=Other Laser Printer Or Label Printers In The DC=
* Mail slots desk in shipping has a Zebra S4M connected to an HP print box
* DC Supervisors office has a Zebra 2746e connected to a Lexmark print box under the desk using a ribbon to print shelf labels
* The cold chain receiving workstation in Receiving has a Zebra TLP 2844 connected via parallel cable to print item labels
* The desktop that JeffP uses has a parallel cable attached Zebra LP2242 used to print item labels
* The desktop that DexterG uses has a parallel cable attached Zebra TLP2742 used to print item labels
* There is a network attached Zebra GX420t label printer that BarryF uses for item labels
* Under the mezzanine in Receiving there is a Zebra S4M attached to an HP print box that is used to print license plates

=Recommendations For Future Improvement=
* According to Gerald if the Buyers moved items out of the "Tobacco Area" into some other spot in the warehouse then we could take the printers from that picking station and make them spares
* License plate label printer in Receiving may not need to be there because any S4M can print license plates
* The high traffic picking stations have network cabling that can be used to network attach printers without the use of a HP print box, but tracing and terminating those network cables doesn't happen instantaneously.
* Something think about is that Zebra has industrial spec label printers similar to the S4M that can connect via wifi which would greatly simplify the cabling mess in the DC
* If we are to use the Lexmark MS521 printers then we have to have the corresponding cable termination supplies and tools
* Pictures (some blurry) of all the printers mentioned on this page are available on the SuperServer in Tech\Common

=Links To Other Printer Wiki Pages=
* Mobile_belt_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mobile_belt_printers]]
* Lexmark_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Lexmark_printers]]
* Xerox_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Xerox_printers]]
* Zebra_Thermal_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Zebra_Thermal_printers]]

[[Category:Printing]]

Information Systems:Printers And Picking Stations In The DC

2019-06-27T18:36:47Z

Darrenf: /* Ready Spares Printer Fleet Totals */

=Production Printer Fleet Totals=
* Lexmark MS415 invoice printer - 1
* Lexmark T650 invoice printers - 8
* Lexmark T640 invoice printers - 2
* Lexmark T630 invoice printers - 1
* Zebra S4M direct thermal label printers - 20
* Zebra 2746e label printers - 4 plus 1 using a ribbon for a total of 5
* Zebra small label printers - 1(GX420t) 1(LP2242) 1(TLP2742) 1(TLP2844) in Returns and Receiving

=Ready Spares Printer Fleet Totals=
* Lexmark MS521 new in box - 3
* Lexmark T650 spare invoice printers - 2
* Lexmark T640 spare invoice printers - 2
* Lexmark T630 spare invoice printers - 1
* Zebra S4M ready spare - 5
* Zebra 2746e ready spare - 2
* Broken printers of any type ready to be recycled - 2

=Production Picking Stations=
* OTC picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HABA picking stations has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* LargeDown picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Lower90s picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Pharmies picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HomeHealthCare picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* FridgeArea picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* NarcCage picking station has 1 Lexmark MS415 and 2 Zebra S4M all connected via parallel cable to an HP print box (Lexmark MS410 also network attached)
* Candy picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* SalesArea picking station has 1 Lexmark T650 and 2 Zebra 2746e all connected via parallel cable to an HP print box
* TobaccoArea picking station has 1 Lexmark T630 and 2 Zebra 2746e all connected via parallel cable to an HP print box (yes they still pick here)

=Other Laser Printer Or Label Printers In The DC=
* Mail slots desk in shipping has a Zebra S4M connected to an HP print box
* DC Supervisors office has a Zebra 2746e connected to a Lexmark print box under the desk using a ribbon to print shelf labels
* The cold chain receiving workstation in Receiving has a Zebra TLP 2844 connected via parallel cable to print item labels
* The desktop that JeffP uses has a parallel cable attached Zebra LP2242 used to print item labels
* The desktop that DexterG uses has a parallel cable attached Zebra TLP2742 used to print item labels
* There is a network attached Zebra GX420t label printer that BarryF uses for item labels
* Under the mezzanine in Receiving there is a Zebra S4M attached to an HP print box that is used to print license plates

=Recommendations For Future Improvement=
* According to Gerald if the Buyers moved items out of the "Tobacco Area" into some other spot in the warehouse then we could take the printers from that picking station and make them spares
* License plate label printer in Receiving may not need to be there because any S4M can print license plates
* The high traffic picking stations have network cabling that can be used to network attach printers without the use of a HP print box, but tracing and terminating those network cables doesn't happen instantaneously.
* Something think about is that Zebra has industrial spec label printers similar to the S4M that can connect via wifi which would greatly simplify the cabling mess in the DC
* If we are to use the Lexmark MS521 printers then we have to have the corresponding cable termination supplies and tools
* Pictures (some blurry) of all the printers mentioned on this page are available on the SuperServer in Tech\Common

=Links To Other Printer Wiki Pages=
* Mobile_belt_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mobile_belt_printers]]
* Lexmark_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Lexmark_printers]]
* Xerox_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Xerox_printers]]
* Zebra_Thermal_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Zebra_Thermal_printers]]

[[Category:Printing]]

Information Systems:Printers And Picking Stations In The DC

2019-06-27T18:33:14Z

Darrenf: /* Links To Other Printer Wiki Pages */

=Production Printer Fleet Totals=
* Lexmark MS415 invoice printer - 1
* Lexmark T650 invoice printers - 8
* Lexmark T640 invoice printers - 2
* Lexmark T630 invoice printers - 1
* Zebra S4M direct thermal label printers - 20
* Zebra 2746e label printers - 4 plus 1 using a ribbon for a total of 5
* Zebra small label printers - 1(GX420t) 1(LP2242) 1(TLP2742) 1(TLP2844) in Returns and Receiving

=Ready Spares Printer Fleet Totals=
* Lexmark MS521 new in box - 3
* Lexmark T650 spare invoice printers - 2
* Lexmark T640 spare invoice printers - 2
* Lexmark T630 spare invoice printers - 1
* Zebra S4M ready spare - 5
* Zebra 2746e ready spare - 2

=Production Picking Stations=
* OTC picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HABA picking stations has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* LargeDown picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Lower90s picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Pharmies picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HomeHealthCare picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* FridgeArea picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* NarcCage picking station has 1 Lexmark MS415 and 2 Zebra S4M all connected via parallel cable to an HP print box (Lexmark MS410 also network attached)
* Candy picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* SalesArea picking station has 1 Lexmark T650 and 2 Zebra 2746e all connected via parallel cable to an HP print box
* TobaccoArea picking station has 1 Lexmark T630 and 2 Zebra 2746e all connected via parallel cable to an HP print box (yes they still pick here)

=Other Laser Printer Or Label Printers In The DC=
* Mail slots desk in shipping has a Zebra S4M connected to an HP print box
* DC Supervisors office has a Zebra 2746e connected to a Lexmark print box under the desk using a ribbon to print shelf labels
* The cold chain receiving workstation in Receiving has a Zebra TLP 2844 connected via parallel cable to print item labels
* The desktop that JeffP uses has a parallel cable attached Zebra LP2242 used to print item labels
* The desktop that DexterG uses has a parallel cable attached Zebra TLP2742 used to print item labels
* There is a network attached Zebra GX420t label printer that BarryF uses for item labels
* Under the mezzanine in Receiving there is a Zebra S4M attached to an HP print box that is used to print license plates

=Recommendations For Future Improvement=
* According to Gerald if the Buyers moved items out of the "Tobacco Area" into some other spot in the warehouse then we could take the printers from that picking station and make them spares
* License plate label printer in Receiving may not need to be there because any S4M can print license plates
* The high traffic picking stations have network cabling that can be used to network attach printers without the use of a HP print box, but tracing and terminating those network cables doesn't happen instantaneously.
* Something think about is that Zebra has industrial spec label printers similar to the S4M that can connect via wifi which would greatly simplify the cabling mess in the DC
* If we are to use the Lexmark MS521 printers then we have to have the corresponding cable termination supplies and tools
* Pictures (some blurry) of all the printers mentioned on this page are available on the SuperServer in Tech\Common

=Links To Other Printer Wiki Pages=
* Mobile_belt_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mobile_belt_printers]]
* Lexmark_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Lexmark_printers]]
* Xerox_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Xerox_printers]]
* Zebra_Thermal_printers[[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Zebra_Thermal_printers]]

[[Category:Printing]]

Information Systems:Printers And Picking Stations In The DC

2019-06-27T18:32:10Z

Darrenf:

=Production Printer Fleet Totals=
* Lexmark MS415 invoice printer - 1
* Lexmark T650 invoice printers - 8
* Lexmark T640 invoice printers - 2
* Lexmark T630 invoice printers - 1
* Zebra S4M direct thermal label printers - 20
* Zebra 2746e label printers - 4 plus 1 using a ribbon for a total of 5
* Zebra small label printers - 1(GX420t) 1(LP2242) 1(TLP2742) 1(TLP2844) in Returns and Receiving

=Ready Spares Printer Fleet Totals=
* Lexmark MS521 new in box - 3
* Lexmark T650 spare invoice printers - 2
* Lexmark T640 spare invoice printers - 2
* Lexmark T630 spare invoice printers - 1
* Zebra S4M ready spare - 5
* Zebra 2746e ready spare - 2

=Production Picking Stations=
* OTC picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HABA picking stations has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* LargeDown picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Lower90s picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Pharmies picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HomeHealthCare picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* FridgeArea picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* NarcCage picking station has 1 Lexmark MS415 and 2 Zebra S4M all connected via parallel cable to an HP print box (Lexmark MS410 also network attached)
* Candy picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* SalesArea picking station has 1 Lexmark T650 and 2 Zebra 2746e all connected via parallel cable to an HP print box
* TobaccoArea picking station has 1 Lexmark T630 and 2 Zebra 2746e all connected via parallel cable to an HP print box (yes they still pick here)

=Other Laser Printer Or Label Printers In The DC=
* Mail slots desk in shipping has a Zebra S4M connected to an HP print box
* DC Supervisors office has a Zebra 2746e connected to a Lexmark print box under the desk using a ribbon to print shelf labels
* The cold chain receiving workstation in Receiving has a Zebra TLP 2844 connected via parallel cable to print item labels
* The desktop that JeffP uses has a parallel cable attached Zebra LP2242 used to print item labels
* The desktop that DexterG uses has a parallel cable attached Zebra TLP2742 used to print item labels
* There is a network attached Zebra GX420t label printer that BarryF uses for item labels
* Under the mezzanine in Receiving there is a Zebra S4M attached to an HP print box that is used to print license plates

=Recommendations For Future Improvement=
* According to Gerald if the Buyers moved items out of the "Tobacco Area" into some other spot in the warehouse then we could take the printers from that picking station and make them spares
* License plate label printer in Receiving may not need to be there because any S4M can print license plates
* The high traffic picking stations have network cabling that can be used to network attach printers without the use of a HP print box, but tracing and terminating those network cables doesn't happen instantaneously.
* Something think about is that Zebra has industrial spec label printers similar to the S4M that can connect via wifi which would greatly simplify the cabling mess in the DC
* If we are to use the Lexmark MS521 printers then we have to have the corresponding cable termination supplies and tools
* Pictures (some blurry) of all the printers mentioned on this page are available on the SuperServer in Tech\Common

=Links To Other Printer Wiki Pages=
* [[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mobile_belt_printers]]
* [[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Lexmark_printers]]
* [[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Xerox_printers]]
* [[http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Zebra_Thermal_printers]]

[[Category:Printing]]

Information Systems:Printers And Picking Stations In The DC

2019-06-27T18:26:31Z

Darrenf: Created page with "=Production Printer Fleet Totals= * Lexmark MS415 invoice printer - 1 * Lexmark T650 invoice printers - 8 * Lexmark T640 invoice printers - 2 * Lexmark T630 invoice printers -..."

=Production Printer Fleet Totals=
* Lexmark MS415 invoice printer - 1
* Lexmark T650 invoice printers - 8
* Lexmark T640 invoice printers - 2
* Lexmark T630 invoice printers - 1
* Zebra S4M direct thermal label printers - 20
* Zebra 2746e label printers - 4 plus 1 using a ribbon for a total of 5
* Zebra small label printers - 1(GX420t) 1(LP2242) 1(TLP2742) 1(TLP2844) in Returns and Receiving

=Ready Spares Printer Fleet Totals=
* Lexmark MS521 new in box - 3
* Lexmark T650 spare invoice printers - 2
* Lexmark T640 spare invoice printers - 2
* Lexmark T630 spare invoice printers - 1
* Zebra S4M ready spare - 5
* Zebra 2746e ready spare - 2

=Production Picking Stations=
* OTC picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HABA picking stations has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* LargeDown picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Lower90s picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* Pharmies picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* HomeHealthCare picking station has 1 Lexmark T650 and 2 Zebra S4M all connected via parallel cable to an HP print box
* FridgeArea picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* NarcCage picking station has 1 Lexmark MS415 and 2 Zebra S4M all connected via parallel cable to an HP print box (Lexmark MS410 also network attached)
* Candy picking station has 1 Lexmark T640 and 2 Zebra S4M all connected via parallel cable to an HP print box
* SalesArea picking station has 1 Lexmark T650 and 2 Zebra 2746e all connected via parallel cable to an HP print box
* TobaccoArea picking station has 1 Lexmark T630 and 2 Zebra 2746e all connected via parallel cable to an HP print box (yes they still pick here)

=Other Laser Printer Or Label Printers In The DC=
* Mail slots desk in shipping has a Zebra S4M connected to an HP print box
* DC Supervisors office has a Zebra 2746e connected to a Lexmark print box under the desk using a ribbon to print shelf labels
* The cold chain receiving workstation in Receiving has a Zebra TLP 2844 connected via parallel cable to print item labels
* The desktop that JeffP uses has a parallel cable attached Zebra LP2242 used to print item labels
* The desktop that DexterG uses has a parallel cable attached Zebra TLP2742 used to print item labels
* There is a network attached Zebra GX420t label printer that BarryF uses for item labels
* Under the mezzanine in Receiving there is a Zebra S4M attached to an HP print box that is used to print license plates

=Recommendations For Future Improvement=
* According to Gerald if the Buyers moved items out of the "Tobacco Area" into some other spot in the warehouse then we could take the printers from that picking station and make them spares
* License plate label printer in Receiving may not need to be there because any S4M can print license plates
* The high traffic picking stations have network cabling that can be used to network attach printers without the use of a HP print box, but tracing and terminating those network cables doesn't happen instantaneously.
* Something think about is that Zebra has industrial spec label printers similar to the S4M that can connect via wifi which would greatly simplify the cabling mess in the DC
* If we are to use the Lexmark MS521 printers then we have to have the corresponding cable termination supplies and tools
* Pictures (some blurry) of all the printers mentioned on this page are available on the SuperServer in Tech\Common

=Links To Other Printer Wiki Pages=
something

Category:uniPHARM Dictionary

2019-06-13T21:53:44Z

Darrenf: /* uniPHARM esoteric terms */

For the pedant in all of us, this experimental category is intended to define esoteric/in-house terms, or terms which may be technically inaccurate but have established too much of a reputation. Cell Block C is a good example of the former, although we probably don't want to be writing any articles about that... -[[User:Norwinu|norwizzle]] ([[User talk:Norwinu|talk]]) 11:16, 8 December 2016 (PST)

Part of the intent here is that, to preserve search-term hit accuracy (which is the power behind any wiki), 'inaccurate' terms will not be corrected when discovered. Users are too accustomed to these terms and will continue to refer to them as such, making it a futile attempt to correct (new tricks, too many old dogs LOL). As well as linking to articles, this page itself will itself have definitions (read below).

=uniPHARM esoteric terms=

* '''Analyzer''': This term is used to describe the ASW data warehouse. Colloquially, it's also 'what InfoManager did' or 'those sales graphs in Infonet'. Analyzer, officially, is merely the Windows-based reporting program developed by IBS that allows business intelligence/power users to extract and analyze data from ASW. The ASW Business Information Warehouse (the data warehouse), is implicated in a lot of reports used in Analyzer, since that entire part of ASW is intended to compile statistics for ease of reporting. Many ASW balance files are also incorrectly referred to as 'Analyzer files'. ''But whatever, it's Analyzer. I guess.''

* '''JetForms''': The correct product name is 'JetForm'. Admittedly, 'JetForms' sounds more natural.

* '''Mocha''': Refers to the telnet-based terminal emulator software. Mochasoft is the third-party maker of the software used to access the IBM i server via green-screen and to interact with the programs contained within the system (ASW, ASW Extensions etc.). TN5250 is the name of the program, and it was named as such because 5250 is the terminal communication protocol which it emulates (IBM 5250 terminals). Mochasoft is not affiliated with IBM or ASW.

* '''Infonet''': Referring to Infonet as the company intranet website is accurate. Infonet referred to as all things WebSmart is not. WebSmart is the development platform. Web Orders and Infonet were created with WebSmart. You can do a million other things with WebSmart besides Web Orders and Infonet.

* '''WebSmart''': WebSmart is a modernization/development platform. It was named as such because its core purpose and function was to enable IBM i for the web i.e. create websites in an RPG environment. However, its core function as a language translator of sorts (from RPG to their proprietary language, PML) makes it a very powerful tool that would allow a company to develop almost entirely with WebSmart (i.e. you don't even have to make web apps).

* '''Web Orders''': Web Orders is the name for our B2B (business-to-business) ordering portal/website (where customers log in to order product from uniPHARM). Indeed it is a website, but it also accurately describes the entire system (the collection of programs and files), and is legitimately (although never acknowledged as such) an extension to our ERP.

* '''uniTV''': Refers to the entire platform/system surrounding (and including) the firmCHANNEL signage players at select stores. As of 2015/2016, official use of this term was discontinued, in favor of just 'digital signage'.

* '''Dumps area''': The open area just outside the doors separating the warehouse from the office, next to Southwest Central Fill and the DC supervisors' office. Referred to as such because this used to be where paper orders were dumped, to be picked by order pickers. (I imagine they would also come back here to dump the finished pick lists? Who knows...that's ancient stuff).

* '''Playroom''': The room to the right of the Server Room that is used for storage. The Playroom is also where we have Thursday night mud wrestling by invitation only.

* '''Server Room''': On the second floor, approximately above where the narc cage is. This is where alllllll the magic happens. Carol would also occasionally request to go sit in that room for 10 minutes because it was air conditioned and she needed to cool off.

* '''Cage''': Northeast corner of the building in the warehouse. This room is used for storage and can be considered to be a tech work shop where stuff can be unboxed and or repaired or setup. Please try and keep this space tidy and organized. The peeps put in a lot of work getting this room usable. Please also note that this space is different from the other spot that is called the Cage, which is right near the shipping entrance in the shipping area. Don't try and setup any servers in that cage.

* '''Belt Printer''': This is a wifi connected battery powered printer that is used in Receiving. It does come with a belt clip and is small and light enough to be worn on a belt but the staff in Receiving just have them sitting on their carts to print labels.

* '''Gun''': This is a wifi connected battery powered laser barcode reader with attached LCD screen and keyboard. It is not a weapon but try not to point the laser at your co-workers eyeballs. These devices break on a regular basis and the rubber boot coverings are generally pretty gross. The upside is you can tell your friends that staff at uniPHARM sell drugs and carry around a gun for 8 hours a day. Pretty dope stuff.

* '''ASW''': Always something wrong

* '''IBS''': This was the former name of Iptor and IBS usually caused some amount of irritable bowel syndrome.

* '''DC1''': Distributed Confusion version 1 ... nooooo I'm just kidding its not distributed at all.

* '''Steelnet''': Consulting company that helped the Unity project back in 2003-2005.

* '''Unity Project''': This was like uniPHARM's Apollo program to get a man on the moon. We got an ERP going and changed a lot of business processes and got a lot of very important stuff done and managed to not have any mental breakdowns in the process - this was 2003-2005.

* '''Calgary''': A good faith attempt to get a bunch of shareholder pharmacies and a warehouse going in Calgary/Alberta that ended up being....not.

* '''LILT''': Lotus is like that. It means Domino/Notes/Lotus whatever behaves the way it does just because it does - it's like a 14 year old with bipolar disorder - sometimes its totally normal and then blamo, you're in the psych ward because the Domino server can handle mime properly.

Information Systems:Xerox printers

2019-06-13T21:49:26Z

Darrenf:

uniPHARM uses Xerox printers for office printing (as opposed to Lexmark printers used in the warehouse). These printers are under either lease or managed print service agreements:

{| class="wikitable sortable"
|-
! Printer hostname !! Model !! Location / purpose !! Contract type
|-
| reccopier || Xerox VersaLink B405DN || Receiving area office || Managed Print Services
|-
| rtncopier || Xerox VersaLink B405DN || Returns department || Managed Print Services
|-
| elaho || Xerox Versa C60 || 2nd-floor printing alcove || Lease
|-
| stein || Xerox AltaLink B8055 || 1st-floor [[:Category:uniPHARM Dictionary|dumps area]] || Lease
|}

==Support info==

* Contact number for leased printers (Xerox One Number): 1-800-275-9376
* Contact number for printers under Managed Print Services: 1-866-487-4239
* Contract end date: December 2023 (?)

==Old printers==
This section is for reference purposes. These printers are no longer being used.

There are 4 different Xerox copier models in use as of 2016. The WorkCentre 7775 is located in the printing alcove on the second floor. It uses colour and black powder toner and has an attached finisher that can fold paper. The colour touch screen is used to setup copy, scan and fax options. The 7775 has 2 fax lines connected so that it can fax inbound and outbound at the same time. The 7775 is leased and Xerox provides consumable toner and other supplies as part of the lease cost. Xerox also provides technical support and repairs as part of the lease. When requesting supplies, repairs or support you must provide the serial number which can be found on the copiers admin page.

:[http://elaho.unipharm.local/ Elaho Administration Web GUI]

*admin
*1111

Supplies Phone Number 1-800-275-9376
Support Phone Number 1-800-275-9376

The WorkCentre 5855 is located next the Supervisors Office in the shipping area of the DC. It has no colour capability but it does have the ability to fax inbound/outbound. The supplies the 5855 uses are different from the 7775 and there is a much smaller and less complicated touch screen. The 5855 is also on the same lease program as the 7775.

:[http://stein.unipharm.local/ Stein Administration Web GUI]

*admin
*1111

The WorkCentre 4265 copiers are located in the Receiving Office and the Returns Areas in the DC. They are smaller copiers with fewer paper trays. They both do have a single fax phone line attached and have very similar software capabilities compared to the 5855 and are on the same lease program as the 5855 and 7775.

:[http://reccopier.unipharm.local/ Reccopier Administration Web GUI]
:[http://rtncopier.unipharm.local/ Rtncopier Administration Web GUI]

*admin
*1111

The WorkCentre 6605 copiers are slightly smaller than the 4265s but otherwise perform and behave in the same way. The 6605s were free from Xerox and are NOT on the lease program and toner and supplies DO need to be purchased for the 2 that are used by the payroll administrator and general manager. Both of the 6605s are network and fax attached.

==Xerox Fleet Management==
This is a useful portal site that Xerox provides for leased or managed printers
* https://office.services.xerox.com/FMP/LoginPage.aspx
* Username is darrenf@unipharm.com
* Password is NewVisionIT@2051

==Xerox Device Agent Software===
There is an application from Xerox called the Xerox Device Agent, installed on the WDS virtual machine. This program sends the page meter reads for the copiers in Returns and Receiving to WestX for billing purposes. This application consumes a lot of CPU time. DarrenF tried for 4 months to get Xerox and its developers to figure out why a crappy little program uses so much CPU 24/7. Xerox wasn't willing to acknowledge a problem or allow a conversation to happen with their in-house developers - so the support ticket was closed as unresolved. The Xerox Device Agent software needs to continue to run and report billing data to WestX so if the CPU usage becomes a problem, try to get a new version of the app or open a new ticket with Xerox and prepare for pain.

==Xerox NeckBeard Support==
On June 13th, 2019, a Xerox technician was called onsite to fix the C60 that was not copying paper in the top feeder or from the glass. He determined that the OS was corrupt and did a factory reset of the machine which wiped out all of its configuration settings. He failed to notify anyone that he was doing a factory reset and he failed to create a backup clone file of the configuration to restore. This caused a great deal of inconvenience and extra work that was not necessary if a clone file was available. If this dumbfuckery ever happens again, there are saved clone files on the SuperServer in the following location : \\superserver.unipharm.local\Tech\Logs And Backups\Xerox Copier Clone Files\. In that folder are clone files for all the leased and MPS copiers as of June 2019. The 2 Xerox MFDs that are used by RonG and AngelaC cannot produce clone files. In case you haven't caught on, a clone file is a file that contains all the settings that are configured on a printer, with the purpose of "cloning" itself onto another printer of the same model or the exact same one after some asshat decides to do something idiotic.

[[Category: Printing]]
[[Category: Pages with Contact Information]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:40:07Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
The following chart describes where a virtual machine should be located so as to balance the processing, memory and storage loads evenly among the 3 hosts in the cluster. VM's can be moved around or shuffled from host to host when doing maintenance but this is the preferred layout of which hosts house which VM's. Changes to this chart will be documented in the change log at the bottom of this page.
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03
|-
| XClarity || SuperServer || 3CX Appliance
|-
| Lucy || UWDDC2 || TestServer1
|-
| Mail || VMware vCenter Server Appliance || UbuntuDevVM
|-
| Smithers || XTGUI || Windows7 Ent For Adobe LifeCycle
|-
| UWDDC1 || || WindowsXP Pro HP JetDirect
|-
| WDS || || Windows10 Ent Eval
|-
| || || Windows10 Pro Eval
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:37:26Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03
|-
| XClarity || SuperServer || 3CX Appliance
|-
| Lucy || UWDDC2 || TestServer1
|-
| Mail || VMware vCenter Server Appliance || UbuntuDevVM
|-
| Smithers || XTGUI || Windows7 Ent For Adobe LifeCycle
|-
| UWDDC1 || || WindowsXP Pro HP JetDirect
|-
| WDS || || Windows10 Ent Eval
|-
| || || Windows10 Pro Eval
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:36:33Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03 !!
|-
| XClarity || SuperServer || 3CX Appliance ||
|-
| Lucy || UWDDC2 || TestServer1 ||
|-
| Mail || VMware vCenter Server Appliance || UbuntuDevVM ||
|-
| Smithers || XTGUI || Windows7 Ent For Adobe LifeCycle ||
|-
| UWDDC1 || || WindowsXP Pro HP JetDirect ||
|-
| WDS || || Windows10 Ent Eval ||
|-
| || || Windows10 Pro Eval ||
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:33:12Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03 !!
|-
| XClarity || Main || Inbound/Outbound ||
|-
| Lucy || Sub || Outbound only ||
|-
| Mail || Sub || Outbound only ||
|-
| Smithers || Sub || Inbound/Outbound ||
|-
| UWDDC1 || Sub || Inbound/Outbound ||
|-
| WDS || Sub || Inbound/Outbound ||
|-
| Inbound Fax - Stein || Sub || Inbound/Outbound ||
|-
| Angela Chan || Sub || Inbound/Outbound ||
|-
| Ron Gracan || Sub || Inbound/Outbound ||
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:30:59Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03 !!
|-
| Administration Account || Main || Inbound/Outbound ||
|-
| Outbound Fax - Company Main || Sub || Outbound only ||
|-
| Outbound Fax - Purchasing Dept. || Sub || Outbound only ||
|-
| Inbound Fax - Receiving || Sub || Inbound/Outbound ||
|-
| Inbound Fax - Returns || Sub || Inbound/Outbound ||
|-
| Inbound Fax - Elaho || Sub || Inbound/Outbound ||
|-
| Inbound Fax - Stein || Sub || Inbound/Outbound ||
|-
| Angela Chan || Sub || Inbound/Outbound ||
|-
| Ron Gracan || Sub || Inbound/Outbound ||
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:30:20Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03 !!
|-
| Administration Account || Main || Inbound/Outbound ||
|-
| Outbound Fax - Company Main || Sub || Outbound only ||
|-
| Outbound Fax - Purchasing Dept. || Sub || Outbound only ||
|-
| Inbound Fax - Receiving || Sub || Inbound/Outbound2 ||
|-
| Inbound Fax - Returns || Sub || Inbound/Outbound2 ||
|-
| Inbound Fax - Elaho || Sub || Inbound/Outbound2 ||
|-
| Inbound Fax - Stein || Sub || Inbound/Outbound2 ||
|-
| Angela Chan || Sub || Inbound/Outbound ||
|-
| Ron Gracan || Sub || Inbound/Outbound ||
|-
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:29:04Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 15%"|VMHost02 !! style="width: 15%"|VMHost03 !!
|-
| Administration Account || '''Main''' || Inbound/Outbound ||
|-
| Outbound Fax - Company Main || Sub || Outbound only ||
|-
| Outbound Fax - Purchasing Dept. || Sub || Outbound only ||
|-
| Inbound Fax - Receiving || Sub || Inbound/Outbound2 ||
|-
| Inbound Fax - Returns || Sub || Inbound/Outbound2 ||
|-
| Inbound Fax - Elaho || Sub || Inbound/Outbound2 ||
|-
| Inbound Fax - Stein || Sub || Inbound/Outbound2 ||
|-
| Angela Chan || Sub || Inbound/Outbound ||
|-
| Ron Gracan || Sub || Inbound/Outbound ||
|-
|colspan="9"| 1 SRFax requires aliases to have a username > 4 characters, yet the email address cannot be used (reserved for user accounts). So the user names for tomc and timm are 'Tom Chotwanwirach' and 'Tim Matiachuk' respectively. Will they notice? Lol c'mon now, it's fax.
2 Note that while these accounts are of the Inbound/Outbound type (billing-wise), the point is that they are being used for inbound purposes only.
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:25:41Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing
{| class="wikitable"
|-
! style="width: 15%"|VMHost01 !! style="width: 5%"|VMHost02 !! style="width: 9%"|VMHost03 !! Purpose !! style="width: 7%"|Fax number !! Username/login !! Mapped aliases !! Account number !! Notes
|-
| Administration Account || '''Main''' || Inbound/Outbound || Primarily for admin purposes, also the ultimate ''route'' through which outbound faxes travel. || 604-276-8478 || admin@unipharm.com || norwinu, darrenf, jeremym, nancyn || 92531 || nancyn and jeremym aliased to this account for address book management and billing administration, respectively.
|-
| Outbound Fax - Company Main || Sub || Outbound only || Used to group users (majority of staff members) for outbound faxing || 604-276-8478 (routes through Main account) || fax@unipharm.com, or aliased users || Most staff members, with their usernames in AD format (e.g. norwinu; '''Exceptions''': tomc, timm1) || 99797 ||
|-
| Outbound Fax - Purchasing Dept. || Sub || Outbound only || Used to group Purchasing Dept. users for outbound faxing || 604-276-8478 (routes through Main account) || purchasing@unipharm.com, or aliased users || elizag, johnt, judyt, rubys, shannonm, simonl, monicat || 99783 || Separated out so department can keep their own address books, local to the group.
|-
| Inbound Fax - Receiving || Sub || Inbound/Outbound2 || Placeholder for inbound route only; for receiving faxes destined for Receiving department copier (reccopier) || 604-276-5250 || faxreceiving@unipharm.com || None || 99645 ||
|-
| Inbound Fax - Returns || Sub || Inbound/Outbound2 || Placeholder for inbound route only; for receiving faxes destined for Returns department copier (rtncopier) || 604-276-5283 || faxreturns@unipharm.com || None || 99643 ||
|-
| Inbound Fax - Elaho || Sub || Inbound/Outbound2 || Placeholder for inbound route only; for receiving faxes destined for 2nd-floor copier (elaho) || 604-270-9728 || faxelaho@unipharm.com || None || 99639 ||
|-
| Inbound Fax - Stein || Sub || Inbound/Outbound2 || Placeholder for inbound route only; for receiving faxes destined for 1st-floor copier (stein) || 604-270-8537 || faxstein@unipharm.com || None || 99647 ||
|-
| Angela Chan || Sub || Inbound/Outbound || Inbound and outbound; dedicated for use by Angela only || 604-270-2884 || angelac@unipharm.com || None || 99571 || Inbound faxes will print to ''angelacprinter''
|-
| Ron Gracan || Sub || Inbound/Outbound || Inbound and outbound; dedicated for use by Ron only || 604-276-5255 || rong@unipharm.com || None || 99579 || Inbound faxes will print to ''rongprinter''
|-
|colspan="9"| 1 SRFax requires aliases to have a username > 4 characters, yet the email address cannot be used (reserved for user accounts). So the user names for tomc and timm are 'Tom Chotwanwirach' and 'Tim Matiachuk' respectively. Will they notice? Lol c'mon now, it's fax.
2 Note that while these accounts are of the Inbound/Outbound type (billing-wise), the point is that they are being used for inbound purposes only.
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:23:15Z

Darrenf: /* VM To Host Optimized Layout */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing
{| class="wikitable"
|-
! style="width: 15%"|Account Name !! style="width: 5%"|Account type (Main/Sub) !! style="width: 9%"|Billing account type (Inbound/Outbound) !! Purpose !! style="width: 7%"|Fax number !! Username/login !! Mapped aliases !! Account number !! Notes
|-
| Administration Account || '''Main''' || Inbound/Outbound || Primarily for admin purposes, also the ultimate ''route'' through which outbound faxes travel. || 604-276-8478 || admin@unipharm.com || norwinu, darrenf, jeremym, nancyn || 92531 || nancyn and jeremym aliased to this account for address book management and billing administration, respectively.
|-
| Outbound Fax - Company Main || Sub || Outbound only || Used to group users (majority of staff members) for outbound faxing || 604-276-8478 (routes through Main account) || fax@unipharm.com, or aliased users || Most staff members, with their usernames in AD format (e.g. norwinu; '''Exceptions''': tomc, timm1) || 99797 ||
|-
| Outbound Fax - Purchasing Dept. || Sub || Outbound only || Used to group Purchasing Dept. users for outbound faxing || 604-276-8478 (routes through Main account) || purchasing@unipharm.com, or aliased users || elizag, johnt, judyt, rubys, shannonm, simonl, monicat || 99783 || Separated out so department can keep their own address books, local to the group.
|-
| Inbound Fax - Receiving || Sub || Inbound/Outbound2 || Placeholder for inbound route only; for receiving faxes destined for Receiving department copier (reccopier) || 604-276-5250 || faxreceiving@unipharm.com || None || 99645 ||
|-
| Inbound Fax - Returns || Sub || Inbound/Outbound2 || Placeholder for inbound route only; for receiving faxes destined for Returns department copier (rtncopier) || 604-276-5283 || faxreturns@unipharm.com || None || 99643 ||
|-
| Inbound Fax - Elaho || Sub || Inbound/Outbound2 || Placeholder for inbound route only; for receiving faxes destined for 2nd-floor copier (elaho) || 604-270-9728 || faxelaho@unipharm.com || None || 99639 ||
|-
| Inbound Fax - Stein || Sub || Inbound/Outbound2 || Placeholder for inbound route only; for receiving faxes destined for 1st-floor copier (stein) || 604-270-8537 || faxstein@unipharm.com || None || 99647 ||
|-
| Angela Chan || Sub || Inbound/Outbound || Inbound and outbound; dedicated for use by Angela only || 604-270-2884 || angelac@unipharm.com || None || 99571 || Inbound faxes will print to ''angelacprinter''
|-
| Ron Gracan || Sub || Inbound/Outbound || Inbound and outbound; dedicated for use by Ron only || 604-276-5255 || rong@unipharm.com || None || 99579 || Inbound faxes will print to ''rongprinter''
|-
|colspan="9"| 1 SRFax requires aliases to have a username > 4 characters, yet the email address cannot be used (reserved for user accounts). So the user names for tomc and timm are 'Tom Chotwanwirach' and 'Tim Matiachuk' respectively. Will they notice? Lol c'mon now, it's fax.
2 Note that while these accounts are of the Inbound/Outbound type (billing-wise), the point is that they are being used for inbound purposes only.
|}

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:19:48Z

Darrenf:

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=VM To Host Optimized Layout=
thing

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-27T17:18:15Z

Darrenf: /* 2019 */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
# DarrenF upgraded all 3 hosts to ESX version 6.5.0-13635690 on May 24, 2019
# DarrenF upgraded the UEFI and IMM2 and DSA firmware on all 3 hosts on May 24, 2019.
# DarrenF upgraded the VMware Tools install on some VM's that could be rebooted during the day on May 27, 2019 - remaining VM's will be upgraded at a later date.
# DarrenF updated this documentation with information on which VM's should located on each host - May 27, 2019.
[[Category: Virtualization]]

Information Systems:VMWare Production Infrastructure

2019-05-22T22:35:50Z

Darrenf: /* 2019 */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
# DarrenF made a backup of the VCSA on May 17, 2019 and saved the backup file to the SuperServer.
# DarrenF upgraded the VCSA from 6.5.0-22000 to 6.5.0.24000 on May 22, 2019 and finished that with a reboot.
[[Category: Virtualization]]

Information Systems:How To Setup A New Staff Member To Use IBM Notes

2019-05-09T16:39:27Z

Darrenf:

# Go to \\superserver\tech\common\domino\registrations
# Copy any text file with a person’s name into a new text file with the new person’s name
# Open the new text file and replace the existing names and addresses with those for the new person – the semi colons represent empty fields so don’t mess them up
# Save that new text file
# Open the Domino Administrator program
# Make sure you are looking at the “People And Groups” tab
# On the top right side of the window expand the “Tools” button
# Expand the “People” button that appears below the Tools button
# Click on “Register”
# The password is ashes ( Stephen Aberle made that password )
# Click the “Import Text File” button and navigate to the saved text file in step 4
# Put a checkmark in the Advanced option on the left side of the Registration window
# Click the Password Options button and make sure that there is a checkmark next to “Set Internet Password”
# Don’t change the slider for password quality unless there is a tiny bomb icon next to the person’s name in the table
# Click OK on the Password Options window
# Make sure there is a checkmark for “Enable roaming for this person” and for “Create a Notes ID for this person”
# Make sure that the correct first and last name are displayed
# The “Explicit policy” should be none available
# Click on the Mail tab on the left side of the Registration window
# The mail file template needs to be “Mail (R9)”
# The Mail file owner access needs to be changed to Manager
# The quota and quota warning amounts can be changed IF needed otherwise leave at the default numbers
# Click the green checkmark after each setting change
# Click on the ID Info tab on the left side of the Registration window
# Make sure that there is a checkmark for “In Domino Directory” and for “In File”
# Make sure that the path for saving the ID file on the Superserver is valid ( it is by default )
# Click on the Groups tab on the left side of the Registration window and make sure that the new person is assigned the correct groups
# Click the green checkmark to save those changes
# Repeat steps 13 to 28 if there are multiple people in the imported text file from step 11
# Click the Register All button and wait for the process to finish
# Click OK and then DONE when the process is finished

*Install Lotus Notes on computer in multi-user mode IT Dept completes this step
*Log into Windows using the persons profile, example joes ( Joe Shmoe) and the assigned password
*Double-click on the Lotus Notes icon on the desktop
*On the first grey window that appears, click the "Next" button
*In the top white field, type in your full name, example Joseph Shmoe John Doe Jane Smith
*In the bottom white field, type in the following exactly maggie/UPH
*Make sure that there is a checkmark next to "I want to connect to a Domino server"
*Click the "Next" button
*Make sure that "maggie/UPH" is in the top white field
*The pull down menu that says "Select a network type" click the down arrow and then click on TCP/IP
*In the bottom white field type in the following exactly maggie.unipharm.local
*Type in your Lotus Notes password and allow the program to download your email from the server
*A small box should appear with prompting you to pick a location, always click on "Office" and then click OK
*The program will then show one more small box letting you know that Lotus Notes setup is complete, click OK

[[Category: How-to pages]]
[[Category: System Administration]]
[[Category: I.T. Periodic Tasks]]

Information Systems:How To Setup A New Staff Member To Use IBM Notes

2019-05-09T16:36:48Z

Darrenf: Created page with "stuff Category: How-to pages Category: System Administration Category: I.T. Periodic Tasks"

stuff

[[Category: How-to pages]]
[[Category: System Administration]]
[[Category: I.T. Periodic Tasks]]

Information Systems:DarrenF's Monthly Manual Tasks

2019-05-09T16:35:15Z

Darrenf:

This page describes what manual tasks DarrenF does at the beginning of each month, usually the first weekday after the first of each month.

* The log files that MDaemon generate on the Mail server need to be moved to the SuperServer. The log files are the daily collection of text files that MDaemon zips up and stores in its log folder. Those zip files can be up to a GB for 30 days so they need to be moved to the Tech folder on the SuperServer so that the hard drive on Mail doesn't fill up. If the hard drive does fill up on the Mail server, MDaemon will stop passing email inbound and outbound.
* The same needs to be done for the MDaemon configuration files which are zipped up in the MDaemon\config folder. They are much smaller than the log files but they do need to be moved to the SuperServer for archival reasons. These config files can be used to revert back to a previous setup of MDaemon. They are useful if configuration changes were made that broke something but you don't want to or can't restore from a tape backup.
* The FTP logs on the Mail server are now much smaller - used to be bigger than a GB per file but they were changed in the summer of 2016 to be per user. If the log files are big and are taking up too much disk space, then they can also be moved to the Tech folder on the SuperServer.
* The Infoblox appliances send their own backups to an FTP account on Mail and although the files are small, they need to be moved to the SuperServer for archival reasons in the Tech folder.
* When an email order is sent to UWD, a copy of the raw email is made by MDaemon and dropped into the EMorder folder on the Mail server. There are a bunch of folders organized by year and month that the email order copies need to be moved into. The reason for this is that a Windows machine will encounter reduced performance when trying to access a folder with tens of thousands of files in it. To avoid this the email order copies from each month are moved out of the drop folder into corresponding month folder inside the year folder. Then that months worth of email order copies is zipped up so that the performance of tape backups are not reduced. The backup software can perform much quicker when it is sending 12 zip files to tape rather than 10,000+ 1 kilobyte files. It's a matter of overhead. Unfortunately, the individual order files are stored in a second location in addition to the EMorder folder so the tidying up of the email copies needs to be done twice. The second location is in the user account folder for uni_orders@unipharm.com. When we decommission email orders completely at some point in the future, this manual tidy up process won't need to be done.
* On the WDS server, Spiceworks creates its own zipped up backup files in a folder within Spiceworks. Those zip files can be deleted as they are on multiple tapes and just take up hard drive space.
* On the SuperServer there are two shared folders for the logs and configuration files that the Sophos firewall creates. The logs are compressed tar files and they need to be moved from the drop folder to the corresponding monthly and yearly sub folders. The configuration files are .abf files and are emailed to webmaster@unipharm.com on a daily basis. Once per month the abf file attachments are stripped out of the emails and archived in the Astaro Backups folder on the SuperServer. The abf files are critical to restoring the configuration of the Sophos firewall if an appliance(s) has a hard drive failure.

[[Category: Miscellaneous I.T.]]

Information Systems:DarrenF's Monthly Manual Tasks

2019-05-09T16:33:07Z

Darrenf:

This page describes what manual tasks DarrenF does at the beginning of each month, usually the first weekday after the first of each month.

*The log files that MDaemon generate on the Mail server need to be moved to the SuperServer. The log files are the daily collection of text files that MDaemon zips up and stores in its log folder. Those zip files can be up to a GB for 30 days so they need to be moved to the Tech folder on the SuperServer so that the hard drive on Mail doesn't fill up. If the hard drive does fill up on the Mail server, MDaemon will stop passing email inbound and outbound.
*The same needs to be done for the MDaemon configuration files which are zipped up in the MDaemon\config folder. They are much smaller than the log files but they do need to be moved to the SuperServer for archival reasons. These config files can be used to revert back to a previous setup of MDaemon. They are useful if configuration changes were made that broke something but you don't want to or can't restore from a tape backup.
*The FTP logs on the Mail server are now much smaller - used to be bigger than a GB per file but they were changed in the summer of 2016 to be per user. If the log files are big and are taking up too much disk space, then they can also be moved to the Tech folder on the SuperServer.
*The Infoblox appliances send their own backups to an FTP account on Mail and although the files are small, they need to be moved to the SuperServer for archival reasons in the Tech folder.
*When an email order is sent to UWD, a copy of the raw email is made by MDaemon and dropped into the EMorder folder on the Mail server. There are a bunch of folders organized by year and month that the email order copies need to be moved into. The reason for this is that a Windows machine will encounter reduced performance when trying to access a folder with tens of thousands of files in it. To avoid this the email order copies from each month are moved out of the drop folder into corresponding month folder inside the year folder. Then that months worth of email order copies is zipped up so that the performance of tape backups are not reduced. The backup software can perform much quicker is sending 12 zip files to tape rather than 10,000+ 1 kilobyte files. It's a matter of overhead. Unfortunately, the individual order files are stored in a second location in addition to the EMorder folder so the tidying up of the email copies needs to be done twice. The second location is in the user account folder for uni_orders@unipharm.com. When we decommission email orders completely at some point in the future, this manual tidy up process won't need to be done.
*There is an installation of Kiwi Syslog on the Thermoprofile server. The log files are large and need to be moved to the Tech folder on the SuperServer. These logs are not reviewed often and it would not be a huge loss if the collection of them were to be stopped, however, in the event of a problem with hardware that does send traps to Kiwi Syslog, there might be the important clues to failures in those logs and that is why they continue.
*On the same Thermoprofile server, Spiceworks creates its own zipped up backup files in a folder within Spiceworks. Those zip files can be deleted as they are on multiple tapes and just take up hard drive space.
*On the SuperServer there are two shared folders for the logs and configuration files that the Sophos firewall creates. The logs are compressed tar files and they need to be moved from the drop folder to the corresponding monthly and yearly sub folders. The configuration files are .abf files and are emailed to webmaster@unipharm.com on a daily basis. Once per month the abf file attachments are stripped out of the emails and archived in the Astaro Backups folder on the SuperServer. The abf files are critical to restoring the configuration of the Sophos firewall if an appliance(s) have hard drive failure.

[[Category: Miscellaneous I.T.]]

Information Systems:How To Upgrade The Power8 HMC

2019-05-08T21:08:13Z

Darrenf:

The Power8 server chassis and its attached disk shelves are managed by an appliance called the Hardware Management Console or HMC. The HMC is considered to be an appliance because it is a highly customized Linux OS provided by IBM on what looks to be a normal but specific version of an x3550 1U rack mounted server. The OS for the HMC is somewhat like the OS for the Sophos firewalls - it's closed off but can take updates and upgrades provided by IBM in their own proprietary format. The HMC can be accessed either from the physical screen and keyboard that is attached in the Server Room or from its web administration page:

* https://hmcpower8.unipharm.local/hmc/connect
* Root Username is hscroot
* Password is NewVisionIT

The best practice method to update the HMC is to login to the above web admin page and in the top right corner of the browser click the down arrow next the round question mark icon to display the current version of the HMC. For example it will show version 9 release 1 service pack 921. Write down those numbers and then log out. Next, go to the IBM website and somehow navigate to what IBM calls "Fix Central". From there you have to navigate a series of drop down menus to get to HMC updates. This part of the procedure is vague on purpose because the IBM website is effing garbage and the site can and will change how it displays the never ending sewer that is IBM software updates. Eventually you might pop out onto the right page (or not) and get a list of available HMC updates. Start by following the bread crumbs to the currently installed version (in this example, 9) and then the release and then the service pack level. You should either see more current service packs or a second or third release. Be sure to read the upgrade path to make sure whatever updates are available from IBM can be installed onto the present version. If at this point you are completely lost and unsure what to look for, the alternative is to call IBM at 1-800-426-7378 and create a new ticket where you give them the serial number of the HMC (front panel of the server) and play dumb about what updates are needed. They will be able to lead you through the IBM website to the correct files.

The files that you do eventually download are going to be ISO files that need to be burned onto a DVD. Blank DVD's are in the cabinet near the 2nd floor copier. Try to write the numerical order i.e. 1 2 3 on the DVD's so that they do get installed correctly into the HMC. Take the DVD's over to the HMC server in the server room. Login at the console and navigate to HMC Management on the left side and click updates. Follow the update wizard and feed in the DVD's to the optical drive when asked to. The HMC will read each DVD and do the update and reboot if needed. If the DVD you feed in has an update out of order, the wizard will display an error saying the update is not compatible with the installed version. If that happens you have to call IBM anyway to figure out what incorrect ISO got downloaded and burned. Yes, I know, this sounds like a pain, welcome to how IBM does things. The upside is you only need to do this at most twice a year and you could probably get away with only doing it once a year. It is necessary to keep the HMC updated because if it lags too far behind the Power8, there will be complications when doing PTFs for Bart. You are doing PTFs for Bart right?

After all the available HMC updates are installed, the DVD's can be garbaged and it is recommended that a backup is taken of the HMC configuration. Because the OS is appliance like, it can't be backed up like a conventional server so again, login to the HMC and navigate to HMC Management and start the backup wizard and get it to create a backup on the FTP server. The backup will take at least a couple of hours but when it is done, move that blob file from the FTP server to SuperServer so that it gets onto a real backup. The file that the HMC generates can be imported in if the HMC suffers a total failure and you have to restore the appliance from the original OS DVD's that should also be available from IBM's Fix Central.

[[Category: How-to pages]]
[[Category: System Administration]]
[[Category: I.T. Periodic Tasks]]

Information Systems:How To Upgrade The Power8 HMC

2019-05-08T21:07:12Z

Darrenf: Created page with "The Power8 server chassis and its attached disk shelves are managed by an appliance called the Hardware Management Console or HMC. The HMC is considered to be an appliance be..."

Information Systems:VMWare Production Infrastructure

2019-05-08T18:56:35Z

Darrenf: /* 2019 */

=Hardware=
uniPHARM purchased a pair of Lenovo x3650 M5 servers in March of 2018 from Anisoft to act as hosts for VMware. The machine type of both servers is 8871-16A and the serial numbers are J121W8C and J121W8D. Both servers do have identical hardware components and as of March 2018, they also have identical and current firmware. Both servers have an Intel Xeon E5-2620 v4 processor populating the first socket. The second socket for both servers is empty. The Xeon has 8 cores and 16 threads. Both machines came with an initial 16GB stick of memory in the first slot. An additional 6 sticks of 8GB were installed into each server so that each has 64GBs of memory. Be aware that the motherboard has specific requirements for where additional memory can be inserted. The numerical order for which memory slots can be used is clearly displayed on the top side lid of the server. If more memory is purchased and installed for these servers, the instructions on the top lid must be followed, or the memory will not be recognized correctly. The memory slots labeled from 13 to 24 on the motherboard cannot be used until the second CPU socket has a processor.

Each server has an ServeRAID M5210 controller that is attached to the motherboard. For each M5210, there is also a RAID5 daughter card that is attached which provides additional capabilities. The M5210 can control up to 24 drives for each server. The initial purchase included 5 drives for each server that are 960GB SSDs. Both of the M5210 controllers are configured with a RAID6 set containing the 5 SSDs. The RAID6 volume can survive the failure of 2 drives before there is data loss. There are no warm or cold spare drives available as of March 2018. The total amount of storage space on each server is 2679GB. The strip size is 256KB. All parameters of the RAID6 set were set to default for the M5210 controller. Each server also has a USB thumb drive that is plugged into the motherboard where the hypervisor software is installed. The thumb drive is 2GB in size and is USB2.0.

Each server has 4 network ports that use the Broadcom NetXtreme chip. Each server has an additional network port that is for dedicated IMM2 access. More on the IMM2 is below. Each server has 2 USB and a VGA port on the front side and the back side. The purchased configuration of each server did not include any riser cards for PCIe expansion cards so if there is a need for extra abilities, the riser cards are also required. Each server has 2 power supplies which can share the load as well as take the entire electrical load should 1 fail. Each server has different power sources feeding into each power supply. The top, or number 2 power supply for each server, is fed from the right side PDU in the Power8 rack. The bottom, or number 1 power supply, is fed from the left side PDU. The PDU's are the "power strips" on each side of the rack where the right side gets power from the Leibert UPS and the left gets power from the Power8 UPS at the bottom of the rack. This all means that in the event of a Hydro power failure, the servers will stay up and operational even if one out of two UPS's fails. Both servers will immediately power off if there is a Hydro power failure AND BOTH UPS's also fail or run out of battery power.

Both servers are located in the Power8 rack and are 2U in size. The top server is at rack unit 18 and the bottom server is at rack unit 16. Both servers can be pulled out on the rack rails and serviced while powered on. The hard drives and power supplies are hot swappable but the memory is not. Neither server has a CD-ROM drive so if a disc needs to be used, the only option is to plug in a USB optical drive.

==Hardware - Network Port Map==
This list shows the layout of the network cables that connect the servers to the stacked core network switches in the Server Room.
* Server Room Stacked Switch (1of4 - Top) Port 2 goes to VMHost01 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 14 goes to VMHost02 dedicated IMM2 port
* Server Room Stacked Switch (1of4 - Top) Port 3 goes to VMHost01 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 15 goes to VMHost02 Management Port on eth2
* Server Room Stacked Switch (1of4 - Top) Port 4 goes to VMHost01 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 16 goes to VMhost02 vMotion Port on eth3
* Server Room Stacked Switch (1of4 - Top) Port 5 goes to VMHost01 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 17 goes to VMHost01 LAN2 Uplink on eth1
* Server Room Stacked Switch (1of4 - Top) Port 6 goes to VMHost02 LAN1 Uplink on eth0
* Server Room Stacked Switch (1of4 - Top) Port 18 goes to VMHost02 LAN2 Uplink on eth1

==Hardware Purchase Details==
The purchase order number for this project is 8226136 (dated Feb 13, 2018) with a vendor number of 20145 and invoices numbers 19253, 19254, 19256 dated March 14, 2018.

==Hardware Support==
uniPHARM has a hardware maintenance contract with Lenovo to provide 24x7x365 onsite parts and labour with a 4 hour response time for the period of March 6, 2018 to March 5, 2021. A renewal of this maintenance contract is expected in February of 2021 because the expected life span of these servers is 5 to 6 years. Note that "maintenance" is not a good descriptor of the service. If a hardware component fails, then the replacement of that part is provided by Lenovo at no cost and is installed by a Lenovo technician at no cost. If uniPHARM adds in non-Lenovo parts they are not covered by the existing contract. Additional Lenovo branded parts that are installed after initial purchase are covered under the existing contract. If a non-Lenovo part is installed and causes damage to the server, the agreement becomes null and void. The phone number for parts replacement and technical support under this contract is 1-800-426-7378. This contract does not cover any VMware software.

=IMM2=
The Integrated Management Module II is an out of band service used to control the x3650 server hardware. It is comparable to the HMC for the Power8. The IMM2 is on and active and accessible as long as the server has power feeding into the power supplies. If both power supplies are unplugged, the IMM2 is not active and accessible. The IMM2 resides on a small "SystemOnChip" on the motherboard and is served by a webserver within a small Linux OS within the SOC. The IMM2 also has a dedicated network port that is only used by that function. On smaller 1U servers the IMM2 shares the first Broadcom network port.

The IP addresses assigned to each of the 2 IMM2's are 172.30.18.54 and 172.30.18.55. The host names are bmvmhost01.unipharm.local and bmvmhost02.unipharm.local. BM is a hold over from a previous IBM product called "Baseboard Management" and this naming convention is a continuation of that. The username for both IMM2's is adminit and the password is visionit. The IMM2's are not accessible from the public side of the firewall and need VPN access if logging in from outside the local network.

* https://bmvmhost01.unipharm.local
* https://bmvmhost02.unipharm.local

The IMM2 web interface is primarily used to control the hardware, alert for hardware failures and to provide a screen console for the server when no physical screen-keyboard-mouse is attached. This is a critical function for troubleshooting or diagnosing software crashes no matter what OS or hypervisor is installed. The console screen function can be presented using a ActiveX, or Java, or HTML5 app that is served from the IMM2 - no need to install any EXE on a laptop. The Java client works consistently. The IMM2 can power on or off the server and show very detailed information on temperatures, fan speeds, voltages and firmware levels of all components. The IMM2 is also setup to email alerts to I.S. staff when hardware events occur such as a failed hard drive or power supply. The IMM2 is also configured to call home to Lenovo when a hardware component fails in the same manner that the HMC connects to IBM when a Power8 hardware failure occurs.

As of March 2018, the x3650 servers do have the latest available firmware and should not need any firmware updates unless Lenovo requires it for replacement parts. If updated firmware is needed, it can be installed from within the IMM2 web interface.

=vSphere Hypervisor=
The vSphere (ESX) hypervisor is installed on the USB thumb drive that is plugged into the internal motherboard port for each server. The customized Lenovo version of the ESX installer was used as it is pre-compiled with all of the device drivers present in Lenovo branded hardware. Each server, now known as a host, is set to only boot from that USB thumb drive. The hypervisor operating system boots and loads into memory and is thusly ready to house and run virtual machines.
* Server with serial number J121W8C has a host name of vmhost01.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.19 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
and
* Server with serial number J121W8D has a host name of vmhost02.unipharm.local
* The administrator username is root
* The administrator password is NewVisionIT
* The management network is on eth2 which is the third network port on the back of the server
* The management network IP address is 172.30.18.20 subnet 255.255.248.0 gateway 172.30.16.1 and is configured to do DNS lookups to 172.30.18.13 and .14
* The management network is using the default VLAN
The web administration pages to directly access the ESX hypervisor and bypass vCenter are:
* https://vmhost01.unipharm.local
* https://vmhost02.unipharm.local
The above links that go directly to the hypervisor don't need to be used or accessed for day to day administration because all administrative tasks should be done within the vCenter user interface. The above links only need to be accessed if vCenter is unavailable, down or broken.

=vCenter Server Appliance=
During Stage 1 of the VCSA setup, the following settings were used
* The FQDN of the VCSA is vcsa.unipharm.local
* The IP address of the VCSA is 172.30.18.23
* The password for the VCSA is NewVisionIT@2051
* The password requires upper and lower case letters AND a number AND a special character but no spaces are allowed.
* The VCSA has an integrated, as in no external, Embedded Platform Services Controller
* The VCSA was deployed in "Tiny" mode and is using a thin provisioned virtual disk
During Stage 2 of the VCSA setup, on the SSO configuration screen, the following settings were used
* Single Sign-On domain name is vsphere.local
* Single Sign-On user name is administrator
* Single Sign-On password is NewVisionIT@2051
* Site name is uniPHARM
* SSH access was enabled
* The following link was used as a guide for the very confusing and badly designed SSO setup https://esxsi.com/2016/11/16/vcsa65/
Please note that if the VCSA virtual machine needs to be rebooted, it will take a good 5 minutes for the web page to be accessible and you may see a plain text page saying the interface is initializing so be patient as the appliance settles down after a reboot.
==vCenter Management==
* The URL for vCenter Flash site is https://vcsa.unipharm.local/vsphere-client/
* The URL for the vCenter HTML5 site is https://vcsa.unipharm.local/ui
* Username is administrator@vsphere.local
* Password is NewVisionIT@2051
* The URL for the vCenter Appliance Management site is https://vcsa.unipharm.local:5480
* The username is root and the password is NewVisionIT@2051
The virtualization infrastructure is designed according to VMware's best practices. In vCenter, a datacenter has been created and called "uniPHARM Datacenter". It contains a cluster called "uniPHARM Cluster. The cluster contains both hosts and any virtual machines are listed under the hosts. The cluster was created with DRS and HA turned off to begin with, however they can be turned on a later time. Each host has a single datastore and they are named VMHost01DataStore01 and VMHost02DataStore01. Each datastore is the entire RAID6 set of 5 SSDs, totaling 2.62TB of usable space. If, in the future, there is ever a need to add storage, the naming convention should continue with VMHost01DataStore02 or VMHost01DataStore03.

===vCenter Management - vNetwork===
As mentioned above, each host has 4 gigabit network ports. The network configuration for our VMware infrastructure is not complex, by VMware standards but the information below is critical to understanding how it has been designed:
* The first (vmnic0) and second (vmnic1) NICs are used for everyday ordinary network traffic going in or out from or to the virtual machines
* The first (vmnic0) NIC is connected to vSwitch2 and is labeled as "uniPHARM Production LAN" - imagine that an invisible virtual switch lies between the first NIC and the real physical switch in the Server Room
* The second (vmnic1) NIC is initially not configured to do anything as of March 2018 but it can be used for a second Production LAN or some sort of testing LAN or fail over if licensing permits
* The third (vmnic2) NIC is configured as a vmKernel port (172.30.18.19 & 20) connected to vSwitch0 acting as the "Management Network"
* The third (vmnic2) NIC and vSwitch0 are only used as a connection that vCenter uses to control the hosts and its setup is a best practice from VMware
* The fourth (vmnic3) NIC is configured as a vmKernel port (172.30.24.1 & 2) connected to vSwitch1 on VLAN 3 acting as the "vMotion Network" and is only used when a virtual motion needs to move from one host to another
It is CRITICALLY important that ANY vnic, vSwitch or vmKernel configuration changes that are made on one host are precisely repeated on the other host. Having different vSwitch labels, for example, prevents vMotion from succeeding. Our EPK license does not permit distributed switches so we have to do the configuration changes manually to each host.

===VUM VMware Update Manager===
VUM has been configured with a dynamic baseline for the hosts and for the VCSA. The best explanation of how VUM works is this link - https://www.youtube.com/watch?v=X_xGihAfLSo Please note that the safest way to apply patches and updates to the hosts hypervisor is to vMotion all the VM's on one host to another and then to remediate the empty host, then vMotion all the VM's back and do the other host. Since we only have 2 or 3 hosts and don't apply patches very often, this method (while time consuming) results in very little down time and a patched infrastructure. An alternate method is to schedule a Saturday, for example where all the VM's can be gracefully shutdown and then when they are ALL off, use VUM to do a patch cycle of each host one at a time.

If the VCSA itself needs to be updated, see this VMware KB article. https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-6751066A-5D4E-47AC-A6A4-5E90AEC63DAA.html The VCSA should be updated and rebooted BEFORE applying any new patches or versions to a host.

===vCenter Alarm Configuration===
vCenter has been configured to use a small number of alarms that email alerts when certain conditions are encountered. The alarm configuration is done at the VCSA level in the hierarchy:
* "Host connection and Power State alarms when the Connection State is equal to Not Responding and the host is at Standby or Powered Off - then emails DarrenF and NorwinU
* "Host CPU Usage" alarms a warning when CPU is above 75% for 10 minutes and a critical when the CPU is above 90% for 10 minutes - then emails DarrenF and NorwinU
* "Virtual Machine CPU Usage" alarms a warning when a vCPU is above 90% for 20 minutes and alarms critical when vCPU is at 100% for 30 minutes - emails DarrenF and NorwinU
* "Virtual Machine Memory Usage" alarms a warning when a memory is above 90% for 10 minutes and alarms critical when memory is above 95% for 10 minutes - emails DarrenF and NorwinU
* "Datastore Usage On Disk" alarms a warning when datastore usage is above 80% and alarms critical when above 90% - emails DarrenF and NorwinU
* "uniPHARM Alarm Network Unplugged" alarms when a NIC is unplugged - emails DarrenF and NorwinU
* "uniPHARM Alarm VM Powered Off" alarms when a VM is powered off - emails DarrenF and NorwinU

===Content Library===
Within vCenter, a content library called uniPHARM Content Library has been created. The purpose of the library is to store ISO and OVF files that are used when creating new VM's so that an OS can be booted - remember that the VMHosts don't have attached CD-ROM drives. Since creating new VM's is not a daily occurrence, the content library should be empty or mostly empty for the majority of the time in order to not consume storage space that is best served for VM's. The content library can only be used for virtual machines created on VMHost01 because that is the datastore where the content library is located. A newly created virtual machine on the other 2 hosts can't use the content library because it's not on shared storage. Also be aware that IE11 cannot upload ISO files greater than 4GB in size, you have to use a different browser. Please for the love of Bhudda, label what you are uploading to the Content Library so that others know what it is.

=Virtual Machines=
A virtual machine is like a bucket. The bucket contains a simulated processor, memory and a flat file that acts as a hard drive. An OS can be installed inside the bucket and can be given a network connection which is also simulated. The OS can't tell the difference between real hardware and simulated so it behaves normally. The bucket that contains the simulated hardware and the OS install, sits on top of the hypervisor. The hypervisor takes simulated hardware calls and maps them to real physical hardware so that each bucket can get a slice of processor and memory. The real hardware can house many buckets and the contents of each bucket don't mix or interact preserving the containerization. And finally buckets can be moved to different hardware without interrupting the OS contents via hocus pocus magic.
==VCSA==
The vCenter appliance VM was created with all the default settings as they were set by the ISO installer. It currently has 2 vCPUs and is using 1.5GB of memory. It has a very unique way of using 14 different thin provisioned VMDK files that act as its hard drives and is currently only using 25GB of real storage space. The VM hardware version is 10. This VM is very important to safe and healthy operation of our virtualization infrastructure.

==XClarity==
XClarity is a software based virtual appliance from Lenovo that shows hardware inventory and does hardware alerting. XClarity is a much more slimmed down modern packaged that is what IBM Director was supposed to be. The OS for the VM is a Linux something where access by the user is restricted. The VM is using 1 vCPU and 8GB of memory and is thin provisioned for 64GB of storage and is actually using only 12GB of storage. The IP address assigned to the VM is 172.30.18.2 and the URL for the sit is :
* https://xclarity.unipharm.local/ui/login.html
The local login for the above URL is:
* Username - adminit
* Password - NewVisionIT2051
* Active Directory SSO is currently not working in XClarity 1.4
The XClarity 1.4 software appears to be fairly straight forward and is able to talk to all the IMM's for our current small fleet of Lenovo servers. The software is set to email DarrenF and NorwinU if there is a hardware failure on any of the Lenovo servers. These alerts are in addition and perform the same function as the IMM's on each machine and is strictly limited to hardware failures. There is an excellent Android app that talks to XClarity over a VPN connection, however it requires that the XClarity appliance root and subordinate certificates be installed on the phone in order to gain access. This is security overkill but DarrenF did install those certificates on his phone and the app is worth this effort though others are free to disagree. This VM appliance was created from an OVA file on May 9, 2018. As of May 2018, this VM is not being backed up by anything and doesn't really need to be because it can be recreated from the OVA file. This VM is in no way business critical and can be powered off if needed.
* It was discovered after the setup of XClarity 1.4 was completed that there is a 2.0 that will be installed and configured again as a VM appliance at a later date. Both versions are supported by Lenovo but because the software is the free version, uniPHARM has no support contract.
* XClarity cannot talk to the IMM on the old SuperServer hardware because the hardware is not on the XClarity HCL. We are going to continue using SuperServer hardware for Veeam but XClarity cannot monitor that hardware. XClarity also cannot monitor Lenovo desktops or laptops - only Lenovo servers, storage SANs and Lenovo switches.

==Thermoprofile==
This is the first physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 100GB and D: from 250GB to 150GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
This VM is ultimately going to go away because the applications on it (Spiceworks, KiwiSyslog, WDS/MDT need to be migrated onto a freshly made Windows Server 2012R2 VM where the OS license is re-used. The conversion of the Windows Server 2008R2 physical machine is acting as a test case for more critical conversions.
* Thermoprofile VM had its applications (Spiceworks and KiwiSsylog + WDS) moved to a new VM called WDS.unipharm.local on Friday June 8, 2018. The Thermoprofile VM was powered off and removed from the BackupExec jobs. The VM was deleted on June 13, 2018.

==Mail==
This is the third physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and not shrunk
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Mail_Server

==SuperServer==
This is the fourth physical server to be converted into a virtual machine:
* The physical hard drives were thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 100GB, the D: was configured to be 1000GB and the E: was configured to be 500GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:SuperServer

==WindowsXP JetDirect==
DarrenF created this VM a long time ago in VMware Workstation so that the HP JetDirect boxes at each picking station can be controlled, configured and setup correctly. The JetDirect boxes are so old that their web administration pages only work with Microsoft Java, which only exists in a 2002 version of WindowsXP prior to SP1 or SP2. The web administration page of the JetDirect boxes works in the same manner as pages for all the other network attached printers. The admin page for the JetDirect boxes need to have the correct IP address and host name set plus other print and network options. New Windows OS's like Windows 7 and 10 are not able to display the JetDirect web admin page because the version of Java it needs does not exist in newer versions of Windows. Having a VM with an old version of Windows is the most convenient method if someone needs to configure a JetDirect box, otherwise, the VM can be left powered off. This VM is only taking up 16GB of storage space.

==Lucy==
This is the second physical server to be converted into a virtual machine:
* Shrunk the 2 physical hard drives - C: from 150GB to 75GB
* Configured the VM to use 2 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
* The VM does not have the serial ports that the physical machine had installed in its PCI slots

==Smithers==
This is the fifth physical server to be converted into a virtual machine:
* The physical hard drive was thick provisioned and shrunk during the conversion process
* The C: drive was configured to be 150GB
* Configured the VM to use 2 vCPU and 16GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
The physical server that was Smithers, had a red coloured USB licensing dongle that was used for the Kofax scanning software. Prior to the P2V, the licensing dongle was moved to the Gauss scanning station in Accounting. The Gauss software "package" on the Smithers VM, plus both scanning desktops now point to the USB dongle in Accounting for licensing compliance. The Smithers VM has no special USB pass-through because the USB dongle was moved prior to the P2V. For more information on this server, please see the following link: http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:Smithers

==WDS==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install for housing the Windows Deployment Service and Spiceworks and KiwiSyslog. WDS is where desktop and laptop images are stored and downloaded from when a workstation does a PXE boot to install Windows.
* The virtual hard drive was thin provisioned at 70GB
* The C: drive was configured to be 100GB
* Configured the VM to use 1 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==UWDDC1==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct)
http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC3_Domain_Controller

==UWDDC2==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as an Active Directory Domain Controller.
* The virtual hard drive was thin provisioned at 50GB
* The C: drive was configured to be 50GB
* Configured the VM to use 1 vCPU and 4GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13
For more information on Active Directory, see the following link: (Yes, it shows UWDDC3, just ignore that, the documentation is valid and still correct) http://elearning.unipharm.local:8080/mediawiki/index.php/Information_Systems:UWDDC4_Domain_Controller

==XTGUI==
This is a new virtual machine created (not converted) with a fresh Windows Server 2012R2 OS install to act as a server for the Iptor XT GUI.
* The virtual hard drive was thick provisioned at 40GB
* The C: drive was configured to be 40GB
* Configured the VM to use 2 vCPU and 8GB of memory
* Configured the VM to use a VMXNET3 and connect it to the Production LAN
* The VM hardware level is 13

==3CX Server Appliance==
This is a new virtual machine created from an ISO file downloaded from the 3CX website. The ISO is an automated install of Debian 64bit plus the 3CX software. The purpose of this VM is to act as a software PBX intended to replace the physical Toshiba PBX hardware. There is a Windows based version of 3CX, however it was deemed that using a prebuilt VM appliance was a better option.
* The virtual machine is configured with 1 vCPU, 4GB of memory and 50GB of thick provisioned disk space. <-- Woops my bad sorry on the thick provisioned space.
* The virtual machine is using the VMXNET3 network interface on the Production LAN at 172.30.18.27
* As of September 2018, this VM is not being backed up in any way
* The virtual machine has the OpenVM Tools installed
* The virtual machine is located on VMHost03
* There is a special VM network that has been created on VMHost03 (only) so that this VM can use the Shaw coax internet connection. Because of this unique network configuration, this VM cannot be vMotioned as the vSwitch is not present on VMHost01 or 02

=Third Host Used For Testing=
On Monday May 28, 2018 a third host server was added to the VMware cluster. The hardware that was used was previously the "Smithers" server. The machine is a Lenovo x3550 M5 and the machine type is a 5463-AC1 and the serial number is E2ZR351. The server has 2 populated processor sockets and contains Intel Xeon E5-2620v3 processors. It has 32GB of memory and has 2 SSD drives that have been put into a RAID0 array. There are 4 network ports at the back of the server and they are used and setup in the same way as the original 2 hosts. This third host has the same IMM2 and the URL and credentials are below:
* https://bmvmhost03.unipharm.local
* adminit
* abc123
The third host has the same build of ESX as the other 2 hosts and it is also installed on a USB thumb drive but the thumb drive in the third host is not a Lenovo part and is not covered under warranty. The third host is also setup to boot directly from the thumb drive instead of the RAID0 SSD array. The RAID0 array has been configured as VMHost03DataStore01 and is 440GB in size. The root password for the installed hypervisor on the third host is the same compared to the other 2 hosts and the management IP used is 172.30.18.21 and the vMotion IP is 172.30.24.3. The third host has an existing Lenovo maintenance agreement that expires on August 26, 2020.

IMPORTANT INFORMATION: The purpose of the third host is for testing and housing a testing environment. The third host is not meant to permanently house any VMs that are used in production. If we care that a VM gets deleted because the datastore was blown away and recreated - then that VM never should have been on the third host. The third host also has a different version of Intel Xeon processor compared to the other 2 hosts. A 2620v3 versus a 2620v4. This matters for doing a vMotion of a running VM either to or from the third host. A vMotion of a powered on VM to or from the third host cannot happen until EVC mode is turned on for the entire cluster. As of May 2018, EVC mode is disabled because enabling it requires an outage where all the VMs in the cluster are powered off and this linked procedure be followed https://kb.vmware.com/s/article/1013111 . If a VM needs to vMotion to or from the third host, it must be powered off and it must fit within the 440GB datastore.

=VMware Tools=
VMware Tools is a package of tools and drivers that most virtual machines need to have installed within the guest OS. VMware Tools allows the hypervisor to talk and manipulate the virtual machine. The tools package also contains many of the virtual hardware drivers that Windows needs to properly enumerate all the "fake" hardware. For example, if a virtual machine is running Windows7, the OS will load the default VGA driver for the video card because natively Windows can't identify a virtualized video card. Same deal for a virtualized network card, Windows won't and probably shouldn't load a built in Microsoft driver. This is where VMware Tools comes in and provides those drivers for the OS. When creating a virtual machine or converting a physical machine, the VMware Tools installer should be installed as soon as possible. The installer can be triggered from the vSphere web interface. The only exceptions to the rule that VMware Tools needs to be installed is if the virtual machine is an "appliance" or pre-built OVF deployment.

=vCenter Standalone Converter=
The vCenter Standalone Converter is a badly named application that converts real physical computers into virtual machines running on a host. The application is Windows based and is typically installed on an IT laptop or workstation. The software creates a "triangle" where it talks to the source computer and the destination host. To begin with the Converter needs the FQDN of the source computer and local administrator credentials. AD administrator credentials are good enough if they are in the local admin group on the source machine. The Converter then uses a WMI connection to figure out what hardware and OS is present on the source machine. During the "Convert Machine" wizard, the Converter also opens a connection to the VCSA which manages the uniPHARM Cluster. Once the Converter understands the source and destination, the user doing the conversion has a chance to customize certain options like how many vCPUs are in the converted VM and how much memory there is and whether or not the source machine is powered off at the end of the conversion.

At the end of the wizard the Converter creates an empty "shell" virtual machine on the chosen destination host and begins to copy the source hard drive(s) to it. It is important to note that the drive copy is direct from the source to the destination and does not hop to the laptop controlling the conversion. The hard drive copy process is using VSS in the context of converting Windows machines so the VMware best practice is to turn off any non-Microsoft services that are running prior to the conversion process to minimize any chance that VSS can't get a lock on a file. It is also best practice to set the source machine in the convert wizard to power off at the end of the last sync so that the newly created VM can be up and running after the last sync and so that there is no duplicate IP address/name conflict. According to VMware documentation, the same "magic" that makes a vMotion happen is used in the Converter to take running physical machines and turn them into running VMs with only a second or two of pause.

When the Converter is finished copying the hard drive(s) and the last sync, the VM is set to either be up and running or powered off and ready to be powered on for the first time as a VM. In the second scenario, when the converted VM is powered on for the first time, it will recognize a bunch of new hardware. For example, an older IBM server would have a physical Broadcom NIC, when it is converted it will have a VMXNET NIC (and therefore use DHCP). The Converter will inject all the drivers the newly converted VM will need to recognize all the different virtualised hardware. There may be a need to do a reboot for a newly created VM as Windows typically asks for one when new hardware drivers are installed. The person doing the conversion can also choose to have the Converter automatically install the VMware Tools package if needed.

Be aware that the amount of time it takes to convert a physical machine to a VM is limited by network bandwidth. All potential source machines are on the same switch stack as the VMHosts so that's as fast as it is going to be. A source machine that has SSDs would be bottlenecked by a 1GB network connection, however an older slower source server with mechanical hard drives might not be. In any case, there is a hard limit on how fast a 1GB connection can move data. Also be aware that after the conversion process is finished and the VM is up and running, Windows may prompt to re-activate the OS license due to how much "hardware" has just changed. Our servers are using volume licenses so there is no reason a re-activation should fail.

The Converter application is currently installed on DarrenF's laptop, but can be installed on other I.S. laptops and is not tied to any licensing. The Converter is free to use but can't do anything useful without vCenter.

=VMware Licensing And Support=
uniPHARM has purchased a vSphere 6 Essentials Plus Kit that includes vCenter. This means that the license entitles us to have a maximum of 3 hosts each having a maximum of 2 CPU sockets and we are entitled to 1 instance of vCenter. The license is usable forever, but the attached technical support is renewed yearly. The licenses and keys are available through the MyVMware portal at
* www.vmware.com
* Username is darrenf@unipharm.com
* Password is visionit
* Account number 114624681
* Customer number 9027898369
uniPHARM also owns a license for a very old version of VMware Server 2.0 and Workstation 9.0 which are both EOL and not usable. While the current EPK licenses are installed correctly inside vCenter, if vCenter needs to be re-installed, the process is to run a licensing report from MyVMware, which generates a CSV file that is imported into vCenter. The next step is to assign the vCenter license to itself and the vSphere licenses to the hosts. VMware has announced that the End Of General support for vSphere 6.5 is November 15, 2021. A support contract for version 6.5 would not be purchasable after that date and if we wanted support, we would be prompted to upgrade to 6.7 or later. This end of general support date applies to the hypervisor install and the vCenter install. Again, the licenses are perpetual and never expire, the support contract is renewable and does expire.

=Backups=
==Veeam==
As of May 2018, uniPHARM is evaluating Veeam Backup And Replication 9.5 as the platform used to do backups of VMware infrastructure and VM's. This section is mostly a placeholder but the sub sections are full of relevant documentation.
===Hardware===
On December 20, 2010, uniPHARM purchased an IBM x3650 M3 server from Anisoft, with asset number 827. The machine type is 7945-AC1 and the serial number is KQ128AR. This machine (as of May 2018) has no IBM/Lenovo hardware maintenance agreement so if and when it has any sort of hardware failure of any kind, IBM/Lenovo will not come onsite with parts to repair. This machine was used as the "SuperServer" until that OS was turned into a VM. This machine does have an older IMM that can be accessed from the following URL and IP:
* http://bmveeam.unipharm.local
* Username adminit
* Password abc123
* IP 172.30.18.46
Because this machine has a very old IMM, it cannot be managed by XClarity. As of May 2018, this machine has the latest firmware for the IMM and UEFI and hard drive controllers. It contains 2 different hard drive controllers. The ServeRAID M1015 and the ServeRAID M5015. This machine has 16 slots for hard drives, numbered 0 to 15 and currently all the slots are populated with 15K-RPM spinning hard drives. The hard drives in slots 0 and 1 are 136GB drives and 2 through 15 are 300GB drives. The 136GB drives are in a RAID1 set and are used as the bootable C: drive with an install of Windows Server 2012R2. The drives in slots 2 to 7 are in a RAID0 as well as drives 8 to 15 in another RAID0 on the second controller. The M1015 and M5015 cannot create a JOBD or RAID0 across controllers because the controllers don't talk to each other. In order to get the maximum amount of disk storage for Veeam, a striped Dynamic Disk was created in Windows as a layer on top of the 2 RAID0 sets to make 1 large D: that is 3.8TB. A failure of any 1 hard drive in slots 2 through 15 will result in a LOSS OF ALL DATA on the D: drive. The 136GB RAID1 set acting as the OS drive on C: can tolerate the failure of a single hard drive without data loss - a failure of both drives will result in data loss. The above arrangement of hard drives is not a best practice, but is an acceptable risk given the nature of backups and budgets.

This machine has a pair of 1GB network cards that are using QLogic drivers as QLogic bought out Broadcom and when Windows Server 2012R2 was installed in May of 2018, the QLogic drivers were installed, however the chips for the NICs are Broadcom. Only the first NIC is being used on IP 172.30.18.9, however the second NIC can be used in a team to provide a 2GB connection with the core Server Room switches are replaced. The older IMM is using a dedicated port on the back of the machine. This server has 1 out of 2 processor sockets occupied with an 8 core Xeon E5620 and 8GB of memory.

===Veeam Software Installation===
In May of 2018, Veeam Backup And Replication 9.5 was installed. This package also installs the Express version of Microsoft SQL 2012 plus supporting files for SQL. Along with Veeam Backup and Replication, Veeam Enterprise Manager and Veeam ONE were also installed as they come free with the perpetual license. Veeam Enterprise Manager is a web based front end for the Veeam Console and has a subset of functions and features compared to the Console. Veeam ONE is a reporting and BI tool that displays graphs and charts and statistics in regards to a VMware/Veeam environment. While Veeam ONE is included free with the perpetual license, it is geared to much larger environments compared to ours - as in hundreds of hosts and thousands of VMs and lots and lots of storage. The full Veeam Console should be also installed on an IT laptop so administrators can make configuration changes if needed without using remote desktop to the physical server. This is very similar to how BackupExec worked. The Veeam Backup And Replication package needs to use the administrator@vsphere.local account to access the VCSA and any subordinate VM's. This account is critical to successful backups and if the password is changed, that change needs to also be changed within the credential manager inside Veeam. The Active Directory domain administrator account is also used to access the remaining physical servers via the Veeam Agent, so if that password is changed, it also needs to be changed within the credential manager inside Veeam.

===Veeam Licensing===
We will probably get perpetual licensing with renewable annual support
===What Is Veeam Backing Up?===
Veeam will be able to backup the VM's and itself and whatever few remaining physical servers that are still in use.
===Backup Strategy And Scheduled Jobs===
Placeholder something something something.
===How Veeam Uses Tape Hardware===
Placeholder something something something.
===Veeam Technical Support===
Put the terms of the support contract in here plus the phone number and expiry dates.

=Change Log - Try And Record Any Major Configuration Or Software/Hardware Changes To The Infrastructure Here=
==2018==
# Both servers have their UEFI/BIOS set to only boot from the USB thumb drive
# Both servers have their UEFI/BIOS set to use maximum performance in the power settings as per this KB article https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098137
# Initial ESX version installed on March 21, 2018 is 6.5.0 Build 7388607
# VMHost01DataStore01 created as the first datastore
# The local datastores are seen as non-ssd by ESX because they are behind a RAID controller and not part of a vSAN
# VCSA password is NewVisionIT@2051
# vcsa.unipharm.local is at 172.30.18.23
# VCSA virtual machine set to automatically start when the hypervisor on VMHost01 boots
# VCSA added to Active Directory as a computer object for SSO on March 22, 2018, located in the Servers OU
# Configed SSO according to https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcsa.doc/GUID-08EA2F92-78A7-4EFF-880E-2B63ACC962F3.html
# Added DarrenF and NorwinU AD user accounts as being able to log into VCSA web client
# vMotion IP on VMHost01 is 172.30.24.1 IP on VMHost02 is 172.30.24.2
# 8 minutes to vMotion 60GB Win7 cagepc as a test P2V on March 23, 2018 with no VLAN
# Created content library and uploaded the xclarity OVF and 3CX ISO files March 23, 2018
# Added VLAN 5 to vMotion Network - NorwinU configed physical switch to use VLAN 5 on ports 4 and 16 on Stacked Switch 1of4 March 23, 2018
# Changed VLAN 5 to 3 for some reason and changed the TCPIP stack for the vMotion Network for optimization
# Added DarrenF and NorwinU AD accounts as permitted Administrators to the entire VCSA hierarchy and got SSO working correctly
# Configured VUM to scan for updates each Monday of every week and email DarrenF
# Ran a VUM Remediation to install Spectre VIBs via the VUM wizard both hosts now at 6.5 build 7967591 March 26, 2018
# P2Ved the Thermoprofile physical server onto VMHost01 March 26, 2018
# Set the Thermoprofile VM to autostart on VMHost01 March 28, 2018
# Tweeked CPU usage alarm to have more time at 90 and 100% - updated documentation April 3, 2018
# P2Ved the Lucy physical server onto VMHost01 April 7, 2018
# P2Ved the Mail physical server onto VMHost01 April 15, 2018
# Logged into the VCSA PSC Controller and set the root password to never expire instead of expiring every 90 days April 25, 2018
# Set the administrator@vsphere.local account to expire its password every 9999 days April 25, 2018
# Turned on the BASH shell for the VCSA PSC and set the correct time zone for NTP April 25, 2018
# Downloaded the VMware trusted root CA certificate from https://vcsa.unipharm.local and installed that into the Trusted Root store on DarrenF's laptop so that the webclient is trusted by IE11 and makes uploading of ISO files to the Content Library possible using an SSO login May 3, 2018
# P2Ved the SuperServer physical server onto VMHost02 May 5, 2018
# Uploaded the ISO for Windows Server Standard 2012R2 to the Content Library May 7, 2018
# Created the XClarity VM appliance from an OVA file in the Content Library - May 9, 2018
# Created a VM for NancyN to use for Adobe LifeCycle - see the notes pane in the Flash client for details on this otherwise undocumented VM - May 16, 2018
# P2Ved the Smithers physical server onto VMHost01 May 19, 2018
# Added hardware that used to be the Smithers server as a third host to the cluster (VMHost03) May 28, 2018
# VMware support ticket 18814250705 for how to enable EVC mode opened and then closed with no changes because enabling EVC requires all VMs to be off May 30, 2018
# VMware support ticket 18815369505 for how to get a database backup from the VCSA to FTP opened May 30, 2018
# New VM created called WDS.unipharm.local with the purpose of housing the WDS feature and Spiceworks and KiwiSyslog - June 8, 2018
# New VMs created called UWDDC1 and UWDDC2 to act as Active Directory Domain Controllers - June 11, 2018
# Thermoprofile virtual machine deleted from the datastore on June 13, 2018
# XClarity virtual appliance deleted and then recreated using the version 2.0 OVA file
# Created new virtual machine called XTGUI with the purpose of it being used as a server for the Iptor DC1 GUI - June 21, 2018
# Created an internal backup of the VCSA and uploaded that to ftp.unipharm.com - August 27, 2018
# Uploaded the VCSA upgrade ISO (6.5.0.22000-9451637) to VMHOST01DATASTORE01 - August 27, 2018
# Upgraded the VCSA to 6.5 U2C build 9451637- August 27, 2018
# Upgraded ESX hypervisors on hosts to 6.5 U2C build 9298722 - September 1, 2018
# Upgraded all VMware Tools installed in all Windows VM's to 10305 - September 1, 2018
# Changed configuration setting "VMkernal.BootHyperthreadingMitigation" from False to True on VMHost03 due to its older processor - September 5, 2018
# Cleared BIOS event logs on VMHost03 and undid the hyperthreading changes from above - September 5, 2018
# Created a new VM called 3CX Server Appliance from an ISO in the Content Library to be used as a PBX - September 26, 2018
# Created new virtual port groups on vSwitch2 on each host to enable VMs to be on Alt-Guest VLAN. The port group is tagged to VLAN 2. - September 28, 2018
# vMotion 3CX appliance to vmhost03 where I can play with it. -17:05, 28 September 2018 (PDT)
# Increased hard drive from 70GB to 100GB on the WDS virtual machine to accommodate new desktop images - October 19, 2018
# Created TestServer1 on VMHost03 - a Windows 2016 server for generic testing purposes. NetStore and TLAForm will be installed on it for testing with DC1.
# Created virtual switch VoIP LAN (VLAN 6). - November 29, 2018
# 3CX Server VM reinstalled and connected to VoIP VLAN. - November 29, 2018
# Created UbuntuTestVM VM, as a generic test VM that can be connected to different VLANs for various network testing / diagnostics purposes. - November 29, 2018
==2019==
# DarrenF created 2 VMs for Windows10 testing on April 4, 2019. The VM's are on VMHost02.
# DarrenF removed an old snapshot for the XTGUI Server and for the vCenter VM on May 8, 2019.
[[Category: Virtualization]]

Information Systems:Sophos UTM Appliance

2019-05-08T18:22:07Z

Darrenf:

=Hardware Appliances Purchased In April Of 2018=
Got new SG210 appliances

=Licensing For Appliances=
Link to https://myutm.sophos.com/
* Username is darrenf@unipharm.com
* Password is #@$Devil505

=Technical Support=
The number is 1-888-767-4679. We have 24x7 phone support entitled from the Premium subscription that was purchased along with the SG210s.

=How To Do A Firmware Update On The Sophos Firewalls=
The SG210 appliance calls home to Sophos on a regular basis and it will email I.S. staff that there is a firmware update available. So we should be clear on what firmware is really referring to here. In Sophos speak, firmware is the highly customized Linux OS install on the firewall appliance that gets a bundled package of RPM files that can be installed from the UTM web administration page. It is a best practice to not install new firmware revisions right after they have been released by Sophos, but rather to wait a week or two or three. The Sophos community forums are a good place to see if the most recent update has some sort of show stopping bug that would affect our network and its security. Sometimes there are bugs but they may be confined to features on the UTM they we don't use and therefore are lest worrisome. If it looks like a firmware update is safe to install, the first step is to log into the web admin GUI and go Management/Uptodate and schedule the firmware update. Because we have two physical appliances working in an active passive high availability pair, a firmware update can safely happen in the middle of a week day. If the firmware update is scheduled for say, 10AM, the install will begin on the slave passive node and when complete that node will reboot and come back up and do a graceful role swap from slave to master. After the active and passive nodes have swapped, the new slave, which was the master, begins its firmware update/reboot/resync cycle. The firmware update only makes changes to the underlying Linux OS, it does not actually change any of the UTM appliance hardware such as the motherboard or NICs. The entire update cycle and role swap back to square can take 20-30 minutes depending on how big the update is. Make sure that you do have a copy of daily firewall configuration backup and a copy of the licensing text file before doing any firmware update to mitigate any scenario where one or both appliances get bricked. Having the configuration backup file and the licensing file and an ISO of the latest OS means you can recover from any disaster quickly without needing to call support.

=QOS=
Below is a link to instructions on how to setup QOS for SIP and VOIP.

https://community.sophos.com/products/unified-threat-management/f/network-protection-firewall-nat-qos-ips/41320/understanding-qos-for-sip-voip-on-the-astaro-sophos-utm

[[Category: Pages with Contact Information]]
[[Category: Networking]]

Information Systems:M-Daemon Mail Server

2019-05-08T17:48:20Z

Darrenf:

=Overview=
MDaemon is a full featured email server that can be considered an alternative to Exchange or Domino or even O365. MDaemon is able to serve as an SMTP, IMAP, POP3 host and can handle many hundreds or thousands of email accounts. The software has a fully featured webmail interface called WorldClient and a web based administration page that mimics most of the configuration options that exist in the GUI. At uniPHARM, the practice is to use the MDaemon GUI to do administrative tasks such as creating new email accounts or watching server activity in real time. MDaemon also acts as a smart host for IBM Domino standing between Domino and the public internet. MDaemon is positioned in this manner because configuring its security settings are much easier compared to Domino. Getting logs from MDaemon is also considerably easier compared to Domino and how the GUI displays configuration options just appears to be more logical and easier to use compared to Domino, plus its safer to expose SMTP, IMAP, POP3 ports from the public internet to MDaemon than allowing all that dirty public traffic onto an interface on the Power8.

Be aware that although the number of email orders flowing from pharmacies through Barracuda to MDaemon has been reduced, there are still some shareholders that send email orders and they do get received by MDaemon and then get forwarded to the Power8 for processing by IMS IMP and IOP. If MDaemon is offline then email flow will stop for everything inbound and outbound. The other big concept to understand is that while MDaemon can stand on its own and do everything related to email, we don't use any of its anti-spam features because Barracuda is responsible for that. MDaemon does scan each inbound and outbound email with its internal anti-virus scanner and can quarantine messages that contain a virus, but its not doing any filtering based on the spaminess of any message. uniPHARM renews the yearly support contract for MDaemon with C&C Software in Ontario each August so that we can call them for support (rare) and be entitled to version upgrades for MDaemon and the Security Plus addon which does the AV scanning. Upgrading MDaemon and Security Plus is always fairly straight forward - stop the services and run each installer. The maker of MDaemon is a company called Alt-N and they put in a lot of effort to make the upgrade installers work safely and consistently. Alt-N is also very aggressive at creating updated versions with bug fixes for the existing code.

If our install of MDaemon does need to be moved to a new VM, the daily configuration backup is going to be absolutely critical because that config backup holds all the settings for MDaemon. According to Alt-N, in order to move the install you first stop all services do an in place upgrade to the latest version, copy the C:\MDaemon folder to the new VM, run the latest installer version again on the new VM and reboot. The new VM would ideally have the same IP and host name and DNS settings as the old VM (which would be off) and that should be enough to get MDaemon up and responding on a new VM. Also, if a move is needed, its necessary to understand that there are shareholder pharmacies that do use our MDaemon server for all of their email needs. Some pharmacies are using O365 or Gsuite but some are dependent on MDaemon and any configuration changes need to have that in mind.

=World Client=
Web Browser Email For Shareholders

MDaemon has a built in web server that is installed on the mail server. The web server is secured with an SSL certificate and the URL is

https://mail.unipharm.com

This website shows up as “World Client”. Think of it as iNotes for MDaemon. Any email account in MDaemon is accessible using this website. The username is the full email address and the password is the password for that email account.

For example

Username test@unipharm.com
Password bed

Only a few shareholders use this service, the majority still use various flavours of Outlook or some other email program. Internet Explorer version 10 and 11 and recent versions of Firefox are required for the World Client website to work. IE 6,7,8,9 will not work.
=Problems=
==Locked Accounts==
MDaemon may automatically accounts based on certain activity patterns. One such rule is the attempt SMTP sending of 250 outbound messages in 30 minutes. This indicates a hijacked account, and requires that the password be changed for the account. Following, this any active SMTP sessions (likely many if this is a botnet hijack) need to be terminated, and this is best done by restarting the SMTP server component of MDaemon.

[[Category: Email]]
[[Category: Servers - Software]]
[[Category: System Administration]]

Information Systems:Purpose And Usage Of File Shares On SuperServer

2019-05-07T19:37:46Z

Darrenf:

This document describes what the purpose and usage is for each file share on SuperServer, hopefully providing insight into why the file shares are laid out the way they are.
* Accounting - This is the department share for the Accounting department. It has a common folder for non-sensitive files and then person folders for each person to manage their own files. This share is restricted to Accounting staff and senior management.
* AcctCommonAR - This share is for accounts receivables and is also restricted to Accounting staff.
* Astaro Backups - This share is where the Sophos firewall configuration backups are stored. The firewall creates an ABF file of its complete configuration and emails that file to DarrenF. That email attachment is then moved to this share so that it makes it onto a backup tape. The configurations are archived so that a previous config can be returned to in the event of a disaster. The share is called Astaro because that's what the firewalls were called before they were bought out by Sophos.
* Astaro Logs - This share is where the firewall's save their log files to. It sure would be nice if they could dump the configuration files to a share as well but no.
* BI - This share was created for the Business Intelligence department which does not exist anymore. JeremyM and JohnT do still use this share.
* Business Development - This share was created for the Business Development department which does not exist anymore. Most of the files in this share are pretty old and are not business critical.
* Buying - This share is heavily used by the Buying department and JohnT and is business critical. The permission restrictions are somewhat relaxed because other staff in other departments sometimes need to get to files located here but while the files are business critical, they are also not sensitive like in Accounting.
* chqMaster - This is an archive folder for the ChequeMaster program. Don't poke at this bear.
* Class 10 Sales - This share is restricted to senior management and contains sensitive sales data.
* Conventions - This share is used by TeresaB and a small number of other staff to plan for yearly conventions.
* Coop Summaries - Not sure what this share is used for but the files are old
* Corporate Relations - This share was used by management staff that are no longer at uniPHARM and most of the files are old and probably not business critical
* Customer Service - This is the department share for Customer Service. Access restrictions are relaxed as there is not a lot of sensitive data.
* DataStorage - This is another share used by management that is no longer at uniPHARM. Lots and lots of files but most are old and this share is mostly just archival data.
* DC1 - This share is being used for the DC1 Upgrade project.
* Distribution Centre - This share is the department share for the Distribution Centre and is used by DC management and the DC supervisors plus Returns and Receiving.
* Employee Changes - This share is used to store files related to new staff, staff leaving or staff changing departments. This share is used by managers and staff involved in staff movements.
* Executive - This share is used by the General Manager and is restricted due to files containing sensitive data and is most likely business critical.
* Executive Confidential - Same
* Finance - This share contains old Accounting data, honestly not sure why this can't be merged with the Accounting share.
* Fire Safety Plans - This share contains files related to emergency situations with relaxed access restrictions.
* Flyer Contracts - This share was and is used by the Buying department and has a lot of old data.
* Forms - This share is very old and is used by management staff for sensitive data.
* FormsGeneral - This share is supposed to be used for forms available for all staff, however in past years it became a dumping ground for files. It does contain a lot of different kinds of data but all staff do have access to FormsGeneral.
* GMP - This share is business critical and contains files on product recalls.
* Health Canada Sales Data Extract - This share is business critical and contains files for Health Canada.
* Home Healthcare - This share is the department share for Home Healthcare which is a subgroup in Buying.
* IFP - This share was used by the Buying department at some point in the past.
* Management - This share is used by management and has access restrictions.
* Medicine Centre - This share has a bajillion files in it and was used by staff in the MC department which does not exist mostly anymore.
* Monthly Sales - This share has restricted access and is used by management, also probably business critical.
* Oldorders - This share has an archive of old emailed orders.
* Payroll - This share has restricted access and contains sensitive and business critical data only accessible by the payroll administrator and senior management.
* Pharmacy Innovation - This share was used by staff that are no longer at uniPHARM.
* Pharmacy Ownership - This share is used by the General Manager and contains sensitive data.
* Photos And Videos - This share is used to store photos and videos that all staff have access to.
* Profiles - This share is where Windows Roaming User Profiles are stored and this share is business critical.
* Quarterly - This share is used by TeresaB for the quarterly newsletter.
* Sales Database Files - This share has old sales data with restricted access.
* Shared User Folders - This share is an unrestricted spot where all staff have read/write access and can save anything here except for sensitive documents.
* Signatures - This share is where signature files used by Lotus Notes are stored. All staff have read access to this share.
* Staging - This share is used as a dumping spot for reports output from the Power8. Call it a temporary landing area.
* Strategic Leadership Team - This share contains the launch codes for uniPHARM's ballistic missile defense shield. Access is restricted.
* Systemi Documentation - This share was the baby infant form of this wiki, ain't it great how we have matured.
* Tech - This share is the department share for the I.S department with access restricted to I.S. staff and senior management.
* Training - This share contains very old files from a training course that does not currently exist.
* uniPHARM Projects - This share contains old archived files from projects done in the past.
* uniTV - This share contains files related to the TV's at pharmacies which is a program that has ended and is no longer being supported.
* UNITY - This share contains many files all related to the Unity project that was completed in 2005.
* uniWikiFiles - This share was supposed to be used to house images for this wiki, not sure what it's being used for now.
* UWD Software - This share is a work in progress but may contain application folders that are linked to GPO's that may or may not be applied to computers connected to AD.

[[Category: System Administration]]

Information Systems:Purpose And Usage Of File Shares On SuperServer

2019-05-07T19:33:53Z

Darrenf:

Information Systems:Purpose And Usage Of File Shares On SuperServer

2019-05-07T19:01:56Z

Darrenf: Created page with "This document describes what the purpose and usage is for each file share on SuperServer, hopefully providing insight into why the file shares are laid out the way they are. *..."

Distribution Centre:After Hours Call Routing For Pharmacies Calling In

2019-05-07T18:04:22Z

Darrenf:

The following is the order of events that are programmed into our phone system to handle what happens when a pharmacy calls uniPHARM after 4:30PM. At 4:30PM, the phone system automatically routes calls from outside the building, from Customer Service to the warehouse supervisors. Outside calls will ring the following extensions - 275 247 248 249 290 287 254 231 245. When those extensions ring anyone can answer the phone and handle the shareholder or customer. When those extensions ring and nobody answers, the outside call will go to the general message voicemail and instructions are below to get to that.
At 11:59PM each weekday except Friday, the phone system will automatically switch outside calls back to Customer Service. On Friday nights, outside calls will continue to be routed to the warehouse extensions because warehouse staff are answering the phones on Sunday.

* "Thank you for calling uniPHARM Wholesale Drugs. Please leave your name, phone number and which pharmacy you are calling from and Customer Service will call you back as soon as possible."

Below are the steps that need to be given to the warehouse supervisors in order for them to retrieve messages left by after hours callers. Remember that this is a completely different voicemail box from each person's own personal voicemail. The supervisors will not be able to tell if there is a message to listen to because there is no indicator light on their desk phones for the general message voicemail box. The IS department recommends that DC supervisors check for messages left on the general message voicemail box at or near cutoff times after 4:30PM.

#Go to any desk phone in the DC or office area
#Press 100
#Press # *
#150 #
#000 #

From there you can listen to the message and then either delete it or not delete it. If the message is NOT deleted, then Customer Service will be able to listen to that message the next morning. This is either useful or not useful depending on the what the customer wants. Hopefully all this clears up a lot of confusion in regards to how the phones work after 4:30PM. I do understand that this is a bit complicated and not as easy to use as it could be so please try and monitor things as best as you can. I do encourage you to call after hours and verify that everything is as described. If it is not please let me know. If a manager decides that the Customer Service call queue needs to remain active past 4:30PM, they need to press the NIGHT button on the phone at the receptionist desk and then 1 on the number pad. This delays the automatic shifting of outside calls to the DC. The "delay" can then be cancelled by again pressing the NIGHT button on the same phone and then 3 on the number pad. The IS department also strongly recommends that CS staff make sure that they are logged out of the call queue at the end of each work day. If someone is still logged into the call queue after 4:30PM or later, the outside caller will wait in line forever instead of being routed to a warehouse extension.

* Q: What is the wording of the voicemail greeting played after 4:30PM?
* A: Thank you for calling uniPHARM Wholesale Drugs please leave your name and phone number and which pharmacy you are calling from. We will call you back as soon as possible.

* Q: How many rings to get to voicemail?
* A: 6

* Q: Where is the blinking message light that shows up when a voicemail is left by an outside caller?
* A: The blinking message light only blinks on Customer Service phones and NOT on any DC extensions

* Q: Can the DC staff listen to voicemail messages left after 4:30PM?
* A: Yes but only if they pretend to be a CS person and login to a CS extension (200, 213, 229, 202, 219) in the same manner that a CS person would if they were doing CS work.

* Q: What happens when an outside caller is speaking to Gordon (for example) and a second outside caller needs to speak to someone?
* A: The second call(er) will ring all the same DC extensions just like the first and the person already on the first call hears a beep on their cordless phone. The second caller can either be picked up by a different person or wait 6 rings to go to voicemail.

* Q: Can an extension in the DC be put in the CS call queue, using Nancy's license, AND still be part of the "ring group" that is currently setup to handle after hours calls?
* A: No

[[Category:Distribution Centre]]

Distribution Centre:After Hours Call Routing For Pharmacies Calling In

2019-05-07T18:03:34Z

Darrenf: Created page with "The following is the order of events that are programmed into our phone system to handle what happens when a pharmacy calls uniPHARM after 4:30PM. At 4:30PM, the phone system..."