< Return to Blog

2019 Homelab: Network Rebuild

By Michael de Silva
Homelab, Infrastructure, Networking, Ubiquity, Unifi, Servers, FreeNAS, pfSense, Routers, Firewall, VMs, Xen-Server, XCP-NG, Citrix

Homelab: Network Infrastructure

This covers a "rebuild" of an existing network, which previously occupied a single sub-net. This is coupled with 10G hardware from Ubiquity and a pfSense router to provide advanced capabilities to various VLANs.

pfSense: Interfaces & VLANs

The following were configured inside pfSense which acts as the primary router/firewall designated pfsense-master.

Hardware & Interfaces

  • Intel 6700K CPU
  • Asus Z170 Prime-A mainboard
    • em0
  • StarTech.com Dual Port PCI Express (PCIe x4) Gigabit Ethernet Server Adapter - 2 Port Network Card - Intel i350 NIC - GbE Network Card (ST2000SPEXI) (Amazon)
    • igb0
    • igb1
  • Intel X550-T2 10G NIC
    • ix0
    • ix1

Networks

Physical LAN:

  • WAN em0 // 1G NIC on mainboard
  • LAN ix0 // 10.10.0.0
  • UniFi // 10.10.1.0
    • UniFi Cloud Key // 10.10.1.3
  • LAN2 igb1 // 10.10.2.0 -- This is purely a convenience 'console' port for troubleshooting any pfSense issues on-site.

pfSense config

The list of VLANs shown previously, have been setup in pfSense and tied to the ix0 (LAN) interface.

Not all the VLANs are currently in use, so only those that are being used have been added as 'interfaces'.

UniFi Cloud Key Gen2+

I decided to use a physical UniFi Cloud Key Gen2+ device as this decouples reliance on the FreeNAS backend, to maintain network continuity in the event of a server outage. The Cloud Key is the only device plugged into interface igb0 on the pfSense box, which is a 1G port on the Startech card.

UniFi & 16XG 10G Switches

The UniFi component in the network has a couple advantages, primarily having a singular control plane (the Cloud Key). The Cloud Key can either be a physical device, or even hosted in the Cloud (AWS/Digital Ocean etc) or a local VM should the need arise. This central interface is used to manage manipulating VLANs tied to the various switch ports.

This is a view from the UniFi Cloud Key Gen2+

Network Topology (Physical)

Given this is a homelab, as the name suggests, it is contained within the 3000sqft space within the condo. ISP (WAN) equipment ties into a patch panel in the Utility room, which is right next to the kitchen. The power distribution panel is also located on the same wall (with RCD and other breakers) -- this is where the pFsense box is located, along with the UniFi 16XG (1) switch.

The workstations area is in the master bed room, which also accomodates the home theatre, and gaming consoles. 16XG (2) sits at the ingest point here.

The server room has 16XG (3) at the ingest point on the DINTEK server rack. 16XG (1) ----> 16XG (2) and 16XG (1) ----> 16XG (3) is over CAT6A cabling, installed by the building developers in late 2017.

Network Topology (Virtual)

pfSense Router/Firewall

The entire network is designed around a master pfSense firewall/router (henceforth, to be referred to as the router) to accomplish various goals. Rather than purchasing a Netgate device, this has been built with older PC components but most importantly with an Intel X550-T2 10G NIC, to allow cross-VLAN 10G traffic.

Due to the nature of firewall rules, even though the UniFi 16XGs supports layer-3 switching, traffic will still need to pass through the pfSense firewall and therefore needs a 10G NIC.

Configuration

  • [X] DNSSEC
    • Precautionary floating rule to block WAN traffic to UDP port 53 for any outgoing non-TLS DNS requests
  • [X] OpenVPN (gateway) - can be specifically tied to any particular VLAN (or net)
  • [X] Firewall rules to prevent access to private (VLAN) networks.

OpenVPN & Firewall rules

Beyond the basic OpenVPN setup in pfSense, here are the extra configuration steps involved to tie specific VLANs to route traffic to the VPN gateway.

First, the VLAN net addresses are set for outbound NAT towards the OpenVPN address. These are the two entries at the very bottom.

Firewall rules: Workstations (VLAN)

The way the firewall rules work, are typically, the very last rule — one to "open" the firewall to allow any and all traffic out from the WORKSTATIONS net source. Reading bottom-up: each higher rule takes precedance, overriding later rules.

In the descriptions I've used the term 'route', although in reality this is a misnomer since the firewall does not do any routing. However, the rule does allow traffic to ultimately "be routed".

  • The 3rd rule from the very bottom - Allows WORKSTATIONS net to the ExpressVPN gateway.
  • The 4th rule from the very bottom - Overrides the previous, and forces all traffic going to the Xbox One, to the default WAN gateway.
  • The rules going top to bottom (the first six) allow traffic from the `WORKSTATIONS net to specific destinations, which are intra/cross VLAN.

Firewall rules: LivingRoom (VLAN)

Similar to the previous example,

  • The last rule is the default "open" rule, allowing any and all traffic leaving the LIVINGROOM net outside the firewall
  • The rule above this overrides it and forces any and all traffic leaving the LIVINGROOM net via the ExpressVPN gateway (as long as the gateway is connected).
  • Should the ExpressVPN gateway connection drop, it will fallback to the default (very last rule).
  • Overriding all of these: LIVINGROOM net cannot connect to any private networks.
  • Overriding further: UAP device is allowed to connect to the Unifi Cloud Key (local traffic)