Tag Archives: tunnel

Site to Site VPN between Google Cloud and pfSense on VMware at home

I’ve always wanted to set up a Site to Site VPN between a cloud provider and my home network. What follows is a guide inspired by Configure Google Cloud HA VPN with BGP on pfSense but customized for a Google Wi-Fi home network and updated with some pfSense changes that I had to figure out.

Home Network

When we built the house in 2015, I set up a 3-pack of original Google Wi-Fi (not the “Nest” version) to use as my router and access points throughout the house. Google Wi-Fi is great – it’s very easy to get started. Once deployed, it can generally be thought of as “set it and forget it”. However, it doesn’t provide all the bells and whistles that some of the more advanced home routers offer, but this can be a blessing in disguise because there is less to fiddle with and potentially mess up. Most importantly, it delivers a reliable experience for the family.

My home lab is a simple Intel NUC with a dual-core Intel Core i3-6100U 2.3 GHz CPU and 32 GB RAM. It runs a standalone instance of VMware ESXi 7. I run a few VMs when I need to, but nothing “production”.

Site-to-Site VPN with Google Cloud

Since switching to a full-time focus on cloud engineering and architecture, one of the things I’ve always wanted to try is to set up a Site-to-Site IPsec VPN tunnel with BGP between my home and a virtual private cloud (VPC) network to better understand the customer experience for VPN configuration and network management.

As I mentioned earlier, Google Wi-Fi is rather basic and doesn’t offer any VPN capability, but it can do port forwarding, and when combined with a virtual appliance, that’s all we really need.

pfSense overview

Since Google Wi-Fi does not have any VPN capabilities, I intend to use a pfSense virtual appliance in ESXi to act as a router for virtual machine clients on an internal ESXi host-only network. The host-only network will have no physical uplinks so the only way out to the Internet or the private cloud network is through the pfSense router.

pfSense will provide DHCP, DNS, NAT, and routing/default gateway services only to the clients on the internal host-only network.

Because of the way it is designed, no other router can sit between Google Wi-Fi and the internet without some loss of functionality, including mesh networking, and we do not want to disturb the other users of the network (family), so we will create an isolated network with pfSense on the ESXi host.

pfSense VM will have two virtual NICs:

  • NIC1 is connected to the “VM Network” and has Internet access through the home network.
  • NIC2 is connected to the internal isolated “host-only” network which does not have any connectivity to the Internet.
Network diagram showing connectivity between the Internet and pfSense running as a virtual machine straddling two networks in the ESXi host on the Intel NUC.

pfSense installation

Netgate has a comprehensive guide on how to install the pfSense virtual appliance on VMware ESXi.

Following are some installation tips that I found to be helpful:

  • Upload the pfSense ISO to an ESXi datastore – don’t forget to unzip it first.
  • When creating a new VM for pfSense on ESXi 7, select Guest OS family “Other” and Guest OS version “FreeBSD Pre-11 versions (64-bit)”
  • VM Hardware:
    • Set CPU to 2
    • Set Memory to 1 GB
    • Set Hard Disk to 8 GB
      • Make sure the SCSI adapter is LSI Logic Parallel
    • Set Network Adapter 1 to the home/internet network, mark it as Connect
    • Add a second Network Adapter for the host-only network, leave it as E1000, mark it as Connect
    • CD/DVD Drive 1 set to Datastore ISO file and browse for the pfSense ISO, mark is as Connect

Boot the VM off the ISO, accept the defaults and let it reboot.

On first boot, the WAN interface will have a DHCP IP from the home network (Google Wi-Fi assigns in the 192.168.86.0/24 range) and the internal-facing LAN interface will have a static IP of 192.168.1.1. If this is incorrect, use the “Assign interfaces” menu item in the console to set which NIC corresponds to WAN and LAN appropriately. Use the ESXi configuration page to find the MAC address of each NIC and which network it is connected to in order to configure them appropriately.

Port-forward IPSec ports to pfSense

After pfSense is installed, we need to port-forward the external Internet-facing IPSec ports on the Google Wi-Fi router to the pfSense VM.

Google has recently relocated management of Google Wi-Fi to the Google Home app. Look for the Wi-Fi area, click the “gear” icon in upper right, select “Advanced networking”, and then “Port management”.

Use the “+” button to add a new rule. Scroll through the IPv4 tab to find the new “pfSense” entry and select it. Verify the MAC address shown is the same as the pfSense VM’s WAN NIC connected to the home network (“VM Network”). Add an entry for UDP 500. Repeat for UDP 4500.

Note: It is not possible to configure port forwarding unless the internal target is online. The Google Home app will only show a list of active targets that are connected to the network. If the pfSense host is not present, verify the VM is powered on and connected to the home network.

Port forwarding rules for inbound UDP/500 and UDP/4500 forwarding to the pfSense NIC1 on the home network

By default, pfSense only allows management access through its LAN interface, so the next step is to deploy a Jump VM with a web browser on the host-only network. Use the VM console to access the Jump VM desktop and launch the browser since it will not be reachable on the home network (in case you wanted to RDP). Verify it has a 192.168.1.0/24 IP. It should also be able to reach the internet but this is not required.

pfSense initial configuration

On the Jump VM, browse to https://192.168.1.1, accept the certificate warning, and log in as admin with password pfsense. Step through the wizard.

Some tips:

  • Set the Hostname and Domain to something different than the rest of the network.
  • Configure WAN interface: Uncheck “Block RFC1918 Private Networks”
  • Set a secure password for admin
  • Select Interfaces | WAN
    • Uncheck “Block bogon networks” if selected
    • Click Save and then Apply

Google Cloud VPN configuration

Use the Google Cloud Console for the following steps:

  • Networking | VPC Networks
    • Create a new VPC network or use an existing one. Should have Dynamic routing mode set to Global.
  • Networking | Hybrid Connectivity | VPN
    • Create a new VPN Connection
      • Classic VPN
      • Select VPC network created earlier
      • Create a new external IP address or use an available one
      • Tunnels – set Remote peer IP address to the home external internet IPv4 address (from home, visit https://whatismyipaddress.com/ and note the IPv4 address)
      • Generate and save the pre-shared key – it is needed for pfSense.
      • Select Dynamic (BGP) routing option and create a new Cloud Router. Set Google ASN to 65000. Create a new BGP session, set Peer ASN (pfSense) to 65001. Enter Cloud Router BGP IP of 169.254.0.1, and BGP peer IP (pfSense) of 169.254.0.2
      • Note the external public IP address of the Cloud VPN.

pfSense IPsec configuration

Use the Jump VM web browser for these steps in the pfSense web interface:

  • System | Advanced | Firewall & NAT tab: Allow APIPA traffic
  • VPN | IPSsec, Add P1
    • Set Remote Gateway to the Google Cloud VPN external public IP recorded previously.
    • Set “My identifier” to be “IP address” and enter the external public IPv4 address of the home network recorded earlier.
    • Enter the Pre-Shared Key generated for the Google Cloud VPN tunnel
      • It may not be possible to paste the key in to the VM console – visit https://pastebin.com and create a new “Burn after reading” paste with the key and then access the paste from the Jump VM to retrieve the key.
    • Set the Phase 1 Encryption Algorithm to AES256-GCM
    • Set Life Time to 36000
  • Save and apply changes
  • Show P2 entries, Add P2
    • Mode: Routed (VTI)
    • Local network: Address, BGP IP 169.254.0.2
    • Remote network: Address, BGP IP 169.254.0.1
    • Protocol: ESP
    • Encryption Algorithms
      • AES, 128 bits
      • AES128-GCM, 128 bits
      • AES192-GCM, Auto
      • AES256-GCM, Auto
    • Hash Algorithms: SHA256
    • PFS key group: 14 (2048 bit)
  • Save, Apply changes
  • Click on Firewall | Rules, select IPsec from along the top, Add a new rule
    • Set Protocol to Any
  • Save rule, Apply changes

pfSense BGP configuration

Go to System | Package Manager, click on Available Packages, search for “frr”. Install “frr”. This will connect out to the Internet to retrieve the packages. Wait for it to complete successfully.

Go to Services | FRR Global/Zebra

  • Global Settings
    • Enable FRR
    • Enter a master password.
    • Set Syslog Logging to enabled and set Package Logging Level to Extended
  • Click on Access Lists along the top
    • Add a new Access List
      • Name: GCP
      • Access List Entries: set Sequence to 0, set Action to Permit, check box for Source Any
      • Click Save
  • Click on Prefix Lists along the top
    • Add a new Prefix List
      • Name: IPv4-any
      • Prefix List Entries: set Sequence to 0, set Action to Permit, check box for Any
      • Click Save
  • Click on BGP along the top
    • Enable BGP Routing
    • Set Local AS to 65001 (GCP Cloud Router was set to 65000)
    • Set Router ID to 169.254.0.2 (GCP Cloud Router was set to 169.254.0.1)
    • Set Hold Time to 30
    • At the bottom, set Networks to Distribute to 192.168.1.0/24
    • Click Save
  • Click Neighbors along the top, add a new Neighbor
    • Name/Address: 169.254.0.1
    • Remote AS: 65000
    • Prefix List Filter: IPv4-any, for both Inbound & Outbound
    • Path Advertise: All Paths to Neighbor
    • Save

Checking status

In pfSense, click on Status | FRR

In the Zebra Routes area, you should see “B>*” entries for subnets in the GCP VPC “via 169.254.0.1” (BGP IP of GCP Cloud Router)

In the BGP Routes area, should see Networks listed for GCP VPC subnets, with Next Hop of 169.254.0.1 (BGP IP of GCP Cloud Router) and Path of 65000 (GCP Cloud Router ASN)

BGP Neighbors should list 169.254.0.1 as a neighbor with remote AS 65000, local AS 65001 and a number of “accepted prefixes” which are the VPC subnets.

Visit the Cloud VPN area in Google Cloud Console, the VPN Tunnel should show Established, and the BGP session should also show BGO established.

Visit the VPC and click on its Routes. There should be one listed for the on-premise pfSense LAN, 192.168.1.0/24 via next hop 169.254.0.2.

Validating connectivity

At this point, VMs in GCP should be able to communicate with VMs in the on-premise pfSense LAN 192.168.1.0/24 network.

Create a GCE instance with no public IP and attach it to the VPC subnet. Make sure firewall rules apply to the instance permit ingress traffic from 192.168.1.0/24 network and permit the appropriate ports and protocols:

  • icmp
  • TCP 22 for SSH
  • TCP 3389 for RDP

Wrapping up

If things are not connecting, double-check everything, but also be sure to check the logs in pfSense and in GCP Cloud Logging. The most frequent issue I encountered was a mismatch of proposals by not selecting the right ciphers for the tunnel, or not setting my identifier properly. Also consider how firewall rules will impact communication.

Finally, the settings outlined here are obviously not meant for production use. I don’t claim to understand BGP any more than what it took to get pfSense working with Cloud VPN, so some of the settings I recommend could be enhanced and tightened from a security perspective. As always, your mileage may vary.