Information Systems:Proposal to migrate to externally-hosted DNS

This has been approved and converted into a project, see this page for implementation details

Overview

Maintenance on the Infoblox appliances will soon require (as of March 2017) renewal for another year. At this time, it is appropriate to consider externally-hosted DNS in lieu of continuing with on-site DNS using these appliances.

Advantages/disadvantages of on-site vs. externally-hosted DNS

It is important to note that whether on-site vs. externally-hosted, the basic function of serving DNS records is the same i.e. the differences are in aspects other than core functionality.

On-site

Pros:

• The biggest pro is the unintended consequence of being a small company and thus, flying under the radar. That means we're arguably less susceptible to DDoS, by simple virtue of us (and our DNS hosting) being too unimportant to hack. Meanwhile, attacks on DNS providers cause massive outages that can make an emphatic point for black-hat hackers. It is not a coincidence that TV shows portraying computer hacking e.g. Mr. Robot focus on DNS-hacking storylines.
• DNS uptime coupled to Web server uptime (for the most part). Arguably the most important function of our DNS hosting is to enable traffic to Web Orders. If our network is down for any reason, DNS likely would be down too, and vice versa. This is not so much an advantage, but a consequence of neutral bearing.
• Vanity of having on-site DNS. When you query NS records for unipharm.com (i.e. who does DNS for unipharm.com?), you will get uni3sys.unipharm.com, indicating we likely do our own DNS (although some external DNS providers also let you do this for extra cost). Again, not really an advantage.
• More "control" over its "physical" operation. On paper, this is technically true, and more control is advantageous in a general sense. It is difficult to elaborate on how this applies in practice though. (There's actually a plug that we can pull?)
• Split-view DNS. This is a big plus for on-site DNS in general, but we don't do anything of this sort. A single DNS server can serve requests internally and externally, and the answers to the queries can be different depending on where you're asking from. For example, orders.unipharm.com can mean 172.30.18.10 for internal users, or 207.102.139.101 for users accessing from the internet. You have the advantage of placing each of the multiple interfaces in a DNS server on different sides of the firewall. Again, we don't do this, and instead we have 2 DNS services - AD and Infoblox to serve internal and external requests, respectively. In other companies where their intranet comprises multiple domains (e.g. if we had host1.van.unipharm.com, and host1.cgy.unipharm.com) split-view is much more important.

Cons:

• No failover. More and more DNS providers are offering failover mechanisms with their DNS service, to ensure uptime for companies with multiple paths to their services (multiple WAN connections that are not bonded). Currently, the DNS records would have to be changed manually.
• High initial cost (virtual appliance itself and the licensing).
• High overhead i.e. costs associated with managing the appliance/virtual appliance. The machine physically exists here so it must be managed. This is self-explanatory, and relative to not having a machine here at all, the magnitude of this point should be fairly obvious.
• Technical inefficiency of this solution. We serve only a handful of DNS records, and they rarely change. Yet the DNS infrastructure we maintain is meant for a much more complex level of operation - one that we will never, ever need.
• Performance. This may not be noticeable, but in terms of sheer performance (i.e. DNS response time, time-to-first-byte etc.), one DNS server cannot compete with the massive CDN (content-delivery networks) and clusters of high-performing servers that an external DNS provider offers.
• Vulnerable to DDoS attacks. While less susceptible to such an attack, if someone were to target our DNS servers, we do not have the resources (advanced network monitoring and security, and manpower) to successfully thwart such an attack. We'd be forced to pull the plug or migrate to our failover WAN line (so basically, running away from the problem). If the attack on Dyn required 4 hours of recovery time, a similar attack on our DNS servers would cripple the network here for much longer.

Externally-hosted DNS

Pros:

• Cost. DNS services have drastically come down in price due to competition. Just as a ballpark, it would cost $4000 for a new Infoblox virtual appliance and $1500 for 3 years of support for a device that at best can be stretched to 5-7 years before best practice necessitates a replacement, compared to USD$60 per year for the cheapest enterprise-level DNS provider.
• Failover service. In the case where our main WAN connection goes down, and assuming the firewall fails over elegantly or we get Sophos Multipath working (i.e. active redundancy), DNS would automatically be modified to point to the different set of IP addresses of our second WAN. This would be a major value added to our existing level of service.
• Redundancy through geographical distribution of DNS servers: any service will have more servers (thousands), and located at different parts of the world.
• Even greater redundancy through multiple DNS providers, still at a lower cost than doing on-site. DNS providers offer primary and secondary DNS services, with the latter being at a much lower cost. I've already detailed a scenario where we could have 2 DNS providers at less than USD$200/year. That would actually avoid being affected by an attack on a single provider. It is worth noting that large, big name companies were exposed by the Dyn outage as using only one DNS provider. That is a testament to how reliable these DNS providers are.
• Decoupling of DNS from web hosting service. If our network was down, external DNS could still point to unipharm.com hosted by Exware. We would be able to post an alert on that website that internal services are down.
• Offload DNS traffic from our network. DNS traffic is not so much a threat to our bandwidth, but rather to network security. By far the highest amount of traffic in our entire network terms of packets is DNS traffic, and most of that is likely invalid anyhow (recursive DNS requests, requests from other countries). The firewall has to do more work.
• Offload management interface to a different network. There is tremendous value in having the management interface on a separate network as well. I've experienced issues getting into the Infoblox (VPN issues, name resolution issues (ironic!), DMZ issues).
• Simplified DNS management. The Infoblox was meant to do much more than just DNS (IPAM, AD integration, DHCP etc.). As a result, the web interface - while robust and abundantly-featured - is tricky to navigate. In addition, the concept of a grid (Infoblox's high-availability implementation) means that there are global and node-local settings to consider, adding further complexity to making otherwise simple record changes or even just looking for stuff.
• Better DDoS protection. Granted Dyn suffered an attack that caused them downtime, but that was really an anomaly compared to the incredibly high uptime observed amongst all DNS providers as a collective. DNS providers are amongst the biggest players in the entire internet and their security mechanisms to prevent DDoS far surpass anything we'd ever think of doing.
• Faster propagation of changes. Whether a manual change or as a result of the failover service, changes to DNS records would propagate faster with externally hosted DNS. They simply have more presence on the internet, and can influence a change much more strongly than a single host can.

Cons:

• Per-instance support billing. Given the nature of DNS as functionally basic but critically important, DNS providers invest in their infrastructure rather than focusing on having dedicated support agents / account reps.
• Dyn outage. We can't ignore that a major DNS provider suffered an attack of that magnitude, and big name sites like Amazon were affected. There is a very simple workaround though, and that is just to use a second DNS provider.
• Limits to number of hosted domains and number of queries. This is only worth stating because Infoblox offers virtually unlimited amounts of hosted domains and queries. However, the quotas with the service we would be acquiring far exceed our needs (and even projected needs of the distant future).

Implications and other considerations for migration

To do:

• Write about the need for a gradual transition
• Current TTL times
• Need to keep uni3sys specified as NS with glue