Christian Hofstede-Kuhn

IT Consultant and Homelab enthusiast.

16 minute read

Isolating services within jails is a cornerstone of FreeBSD security and system administration. Taking this a step further, using VNET (Virtual Network Stack) jails allows each jail to have its own independent network configuration. This guide will walk you through setting up a dedicated VNET jail to act as a Packet Filter (PF) firewall for other application VNET jails, providing robust network segmentation and control.

We’ll configure a FreeBSD host, create a firewall jail (fwjail), an application jail (appjail1), and set up NAT and port forwarding.

Important Note on Placeholders: Throughout this guide, you will see placeholder values like your-freebsd-host.example.com, yourdomain.com, 10.254.253.x, 2a01:4f8:xxx:yyyy:zzz::N, and generic interface names like vtnet0. You must replace these with your actual hostnames, domain names, IP addresses, network prefixes, and physical interface names relevant to your specific environment.

Network Topology

A visual representation of our target setup:

Network Topology

Prerequisites

  • A running FreeBSD system (this guide was developed with FreeBSD 14.2-RELEASE in mind).
  • Root access or sudo privileges on the host.
  • Basic understanding of FreeBSD, networking concepts (IP addresses, subnets, routing), and the PF firewall.
  • Your host’s external IP addresses (IPv4 and IPv6) and gateway information.

I. Host System Configuration

First, we need to prepare the host system by ensuring the necessary kernel modules are loaded and network interfaces are configured.

A. Required Kernel Modules

PF and VNET jails rely on several kernel modules. Ensure they are loaded at boot by adding these lines to your host’s /boot/loader.conf:

# /boot/loader.conf (on the FreeBSD Host)
if_bridge_load="YES" # For network bridges
if_epair_load="YES"  # For virtual Ethernet pairs used by VNET jails
pf_load="YES"        # Packet Filter firewall
pflog_load="YES"     # PF logging interface (for pflog0)

You can load them manually for the current session with sudo kldload <module_name> or reboot after editing loader.conf.

B. Host Network Configuration (/etc/rc.conf)

We’ll configure two bridges on the host:

  • bridge0: The primary bridge connecting to your physical network (external).
  • bridge1: An internal bridge for communication between the firewall jail and application jails.

Here’s an example snippet for your host’s /etc/rc.conf:

# /etc/rc.conf (on the FreeBSD Host)
hostname="your-freebsd-host.example.com" # Set your host's fully qualified domain name

# --- External Network Interface (e.g., vtnet0) ---
# This is your physical interface or the one connected to the internet.
# It will be a member of bridge0.
# Replace vtnet0 with your actual physical interface name (e.g., em0, igb0).
# Options like -tso -vlanhwtso are examples; adjust based on your NIC.
ifconfig_vtnet0="up -tso -vlanhwtso"

# --- Network Bridges ---
cloned_interfaces="bridge0 bridge1"

# Bridge0: External network connectivity
# Add your physical interface (vtnet0) as a member.
# Assign your host's main IP addresses to bridge0.
ifconfig_bridge0="inet6 auto_linklocal -ifdisabled addm vtnet0" # Adds vtnet0, enables link-local
ifconfig_bridge0_descr="Host External Bridge"
ifconfig_bridge0_ipv4="inet 10.254.253.213/24" # YOUR HOST'S EXTERNAL IPv4
ifconfig_bridge0_ipv6="inet6 2a01:4f8:xxx:yyyy:zzz::213/80" # YOUR HOST'S EXTERNAL IPv6

# Bridge1: Internal jail network
# This bridge acts as a Layer 2 switch for internal jails and does not need an IP on the host.
ifconfig_bridge1="up"
ifconfig_bridge1_descr="Jails Internal Bridge"

# --- Default Gateways for the Host ---
defaultrouter="10.254.253.1" # YOUR NETWORK'S IPv4 GATEWAY
ipv6_defaultrouter="2a01:4f8:xxx:yyyy:zzz::1" # YOUR NETWORK'S IPv6 GATEWAY

# --- Enable Jails ---
jail_enable="YES"
jail_parallel_start="YES" # Start jails in parallel
jail_list="" # Start all jails defined in jail_conf_files (or list specific jails)
# Use jail_conf_files to include main config and .conf files in jail.conf.d
jail_conf_files="/etc/jail.conf /etc/jail.conf.d/*.conf"
# jail_set_hostname_allow="NO"; # Recommended for security unless specific needs arise

# --- Other Host Services (examples) ---
sshd_enable="YES"
zfs_enable="YES"

Remember to replace placeholder IPs, network prefixes, and interface names with your actual values. After configuring, you might need to restart networking (sudo service netif restart && sudo service routing restart) or reboot the host.

II. The Firewall Jail (fwjail)

This jail will have two network interfaces: one connected to bridge0 (external) and one to bridge1 (internal). It will run PF to filter and NAT traffic for appjail1.

A. Creating the Firewall Jail Filesystem

First, create a directory for the jail’s root and install the FreeBSD base system.

sudo mkdir -p /usr/jails/fwjail
sudo bsdinstall jail /usr/jails/fwjail
# Or, if you have base.txz: sudo tar -C /usr/jails/fwjail -xvf /path/to/your/base.txz

B. devfs Rules for PF Access

fwjail needs access to the PF device. Edit /etc/devfs.rules on the host and add a new ruleset (e.g., ruleset 7, ensure the number is unique and not used by other system rulesets):

# /etc/devfs.rules (on the FreeBSD Host)
# Add this at the end of the file or integrate into existing custom rulesets.

[devfsrules_fwjail=7]
add include $devfsrules_jail # Includes standard jail devices (typically ruleset 4)
add path 'pf' unhide         # Make /dev/pf accessible
add path 'pflog' unhide      # Make /dev/pflog accessible (for pflog0)

Restart devfs for changes to take effect: sudo service devfs restart.

C. fwjail Configuration (/etc/jail.conf.d/fwjail.conf)

Create a configuration file for fwjail in /etc/jail.conf.d/fwjail.conf on the host.

# /etc/jail.conf.d/fwjail.conf (on the FreeBSD Host)
fwjail {
  $epair_ext_idx = 0; # Unique index for the external epair
  $epair_int_idx = 1; # Unique index for the internal epair

  $host_ext_if = "epair${epair_ext_idx}a";
  $jail_ext_if = "epair${epair_ext_idx}b"; # This becomes ext_if inside fwjail's rc.conf
  $host_int_if = "epair${epair_int_idx}a";
  $jail_int_if = "epair${epair_int_idx}b"; # This becomes int_if inside fwjail's rc.conf

  host.hostname = "fwjail.yourdomain.com"; # Set your firewall jail's hostname
  path = "/usr/jails/fwjail";              # Path to the jail's root filesystem
  mount.devfs;                             # Mount a devfs filesystem in the jail
  devfs_ruleset = 7;                       # Use ruleset defined in /etc/devfs.rules for PF access
  exec.clean;                              # Run in a clean environment
  persist;                                 # Keep jail running even if no primary process
  allow.raw_sockets;                       # Needed for PF, ping, traceroute, etc. Use with caution.

  vnet; # Enable VNET (virtualized network stack)

  # External interface (connects to host's bridge0)
  exec.prestart += "ifconfig epair${epair_ext_idx} create up descr jail_fwjail_ext";
  exec.prestart += "ifconfig bridge0 addm ${host_ext_if}";
  vnet.interface += $jail_ext_if; # Assign epair0b to the jail

  # Internal interface (connects to host's bridge1)
  exec.prestart += "ifconfig epair${epair_int_idx} create up descr jail_fwjail_int";
  exec.prestart += "ifconfig bridge1 addm ${host_int_if}";
  vnet.interface += $jail_int_if; # Assign epair1b to the jail

  # Standard jail startup/shutdown commands
  exec.start += "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";

  # Cleanup commands run on the host after the jail stops
  exec.poststop += "ifconfig bridge0 deletem ${host_ext_if}";
  exec.poststop += "ifconfig ${host_ext_if} destroy";
  exec.poststop += "ifconfig bridge1 deletem ${host_int_if}";
  exec.poststop += "ifconfig ${host_int_if} destroy";
}

D. fwjail Internal Configuration (/usr/jails/fwjail/etc/rc.conf)

Configure the network interfaces and enable services inside fwjail. Edit this file on the host at /usr/jails/fwjail/etc/rc.conf.

# /usr/jails/fwjail/etc/rc.conf
hostname="fwjail.yourdomain.com" # Match host.hostname from jail.conf

# Enable routing and PF
gateway_enable="YES"       # For IPv4 forwarding
ipv6_gateway_enable="YES"  # For IPv6 forwarding (sets net.inet6.ip6.forwarding=1)
pf_enable="YES"            # Enable Packet Filter
pf_rules="/etc/pf.conf"    # Path to PF ruleset file within the jail
pflog_enable="YES"         # Enables the pflog0 interface for logging PF activity

# External Interface (e.g., epair0b, connected to host's bridge0)
# This interface gets an IP on your main external network.
# The name 'epair0b' corresponds to $jail_ext_if from fwjail.conf.
ifconfig_epair0b_name="ext_if" # Alias for use in pf.conf
ifconfig_ext_if="inet 10.254.253.254/24" # UNIQUE IP for fwjail on external network
ifconfig_ext_if_ipv6="inet6 2a01:4f8:xxx:yyyy:zzz::254/80" # UNIQUE GUA for fwjail
defaultrouter="10.254.253.1" # Your main network's IPv4 gateway
ipv6_defaultrouter="2a01:4f8:xxx:yyyy:zzz::1" # Your main network's IPv6 gateway

# Internal Interface (e.g., epair1b, connected to host's bridge1)
# This is the gateway for your application jails.
# The name 'epair1b' corresponds to $jail_int_if from fwjail.conf.
ifconfig_epair1b_name="int_if" # Alias for use in pf.conf
ifconfig_int_if="inet 192.168.100.1/24" # Gateway IP for internal IPv4 jail network
ifconfig_int_if_ipv6="inet6 fd00:cafe:beef:100::1/64" # Gateway ULA for internal IPv6 jail network

# Optional: Enable SSH for managing fwjail directly (ensure pf.conf allows it)
# sshd_enable="YES"

Remember to use unique IP addresses for fwjail’s external interface that are part of your external network.

E. fwjail Firewall Rules (/usr/jails/fwjail/etc/pf.conf)

This is the heart of the firewall. Copy the following into /usr/jails/fwjail/etc/pf.conf on the host. This configuration provides a starting point.

# --- Macros & Port Lists ---
# Macros make the ruleset easier to read and manage.

# Network Interfaces
# These aliases MUST match the 'ifconfig_epairXXb_name="alias"' settings
# in this jail's /etc/rc.conf file.
ext_if = "ext_if" # External interface (e.g., epair10b, connected to bridge0)
int_if = "int_if" # Internal interface for jails (e.g., epair11b, connected to bridge1)

# Internal Network Definitions
# The '$int_if:network' macro dynamically gets the network from the interface configuration.
# Ensure 'int_if' has a correctly configured IP address and netmask/prefix length.
internal_net_v4 = $int_if:network             # e.g., 192.168.100.0/24
internal_net_v6 = "fd00:cafe:beef:100::/64"   # Your Unique Local Address (ULA) prefix for internal jails

# Firewall's External Global Unicast IPv6 Address (GUA)
# For IPv6 NAT, explicitly defining the GUA is more robust than relying on ($ext_if).
# !!! IMPORTANT: Replace "2a01:4f8:xxx:yyyy:zzz::254" with the ACTUAL GUA !!!
# !!! of your fwjail's external interface ($ext_if).                   !!!
fw_ext_ipv6_gua = "2a01:4f8:xxx:yyyy:zzz::254" # EXAMPLE - THIS MUST BE YOUR FWJAIL'S EXT GUA

# Commonly Allowed Outbound Ports for Jails (defined as macros)
# These lists define which destination ports internal jails can connect to.
# Customize these lists based on the services your jails need to access.
allowed_tcp_ports_list = "{ 22, 53, 80, 443, 123 }" # SSH, DNS over TCP, HTTP, HTTPS, NTP
allowed_udp_ports_list = "{ 53, 123 }"             # DNS over UDP, NTP

# --- Options ---
# 'set skip on lo0' disables PF processing for loopback traffic, which is safe and efficient.
set skip on lo0

# 'set block-policy drop' makes the default block action to silently drop packets.
# 'return' would send back a TCP RST or ICMP unreachable, which can be revealing.
set block-policy drop

# 'set loginterface pflog0' directs PF to log to the pflog0 pseudo-interface.
# Ensure pflog0 is created at boot: 'ifconfig pflog0 create up' (usually via pflog_enable in rc.conf).
# Also ensure the pflog.ko kernel module is loaded on the host.
set loginterface pflog0

# 'set state-policy if-bound' binds states to the interface they were created on.
# This is generally a good security practice.
set state-policy if-bound

# Adjust various state timeouts. These are examples and can be tuned.
set timeout { tcp.first 60, tcp.opening 30, tcp.established 86400 } # tcp.established is 24 hours
set timeout { udp.first 30, udp.single 30, udp.multiple 60 }
set timeout { icmp.first 20, icmp.error 10 }
set timeout { adaptive.start 0, adaptive.end 0 } # Disable PF's adaptive state timeouts for predictability

# --- Normalization ---
# 'scrub in all' reassembles fragmented packets and normalizes some TCP options.
# This helps protect against certain attacks and protocol ambiguities.
# 'max-mss 1440' clamps the Maximum Segment Size for TCP to prevent PMTU discovery issues,
# especially common with PPPoE or VPN tunnels that reduce the effective MTU.
scrub in all max-mss 1440

# --- NAT Rules ---
# Network Address Translation rules are processed before filter rules for outgoing packets.

# Outbound NAT for IPv4 traffic from internal jails.
# Packets from $internal_net_v4 going out $ext_if will have their source IP changed
# to the primary IP address of $ext_if.
# 'port 1024:65535' suggests a port range for source port translation; can help some protocols.
nat on $ext_if from $internal_net_v4 to any -> ($ext_if) port 1024:65535

# Outbound NAT for IPv6 traffic (ULA to GUA) from internal jails.
# Packets from $internal_net_v6 going out $ext_if will have their source IP changed
# to the explicitly defined $fw_ext_ipv6_gua.
nat on $ext_if from $internal_net_v6 to any -> $fw_ext_ipv6_gua

# --- Filtering Rules ---
# Filter rules are processed in order. The last matching 'quick' rule wins,
# or the last matching non-'quick' rule wins if no 'quick' rules match.

# Default block policy: block and log everything unless explicitly passed.
# The 'log' keyword causes blocked packets to be logged via pflog0.
block log all

# Anti-spoofing rules: prevent packets with forged source IPs.
# 'quick' ensures these rules are applied immediately if matched, stopping further processing.
# For the internal interface, block any packets claiming to be from outside its configured network.
antispoof quick for $int_if inet
antispoof quick for $int_if inet6
# Consider 'antispoof quick for $ext_if inet' if your external IP is static and simple.
# If you have multiple IPs or aliases on ext_if, this might be too restrictive.

# Allow all traffic on the loopback interface (already handled by 'set skip on lo0' but explicit pass is fine).
pass quick on lo0

# Allow essential ICMP (IPv4) and ICMPv6 types for network diagnostics and operation.
# For IPv4: Allow ping requests, destination unreachables, and time exceeded messages.
pass quick inet proto icmp icmp-type { echoreq, unreach, timex }
# For IPv6: ICMPv6 is critical for IPv6 functioning (Neighbor Discovery, Path MTU Discovery, etc.).
# It's generally recommended to allow all ICMPv6 to avoid breaking IPv6.
pass quick inet6 proto icmp6 all

# Allow traffic originating from the firewall jail itself to go out.
# This allows the firewall to get DNS, updates, NTP, etc.
# ($ext_if) refers to the IP(s) configured on the external interface.
pass out log quick on $ext_if proto tcp from ($ext_if) to any port $allowed_tcp_ports_list keep state
pass out log quick on $ext_if proto udp from ($ext_if) to any port $allowed_udp_ports_list keep state
pass out log quick on $ext_if inet6 proto icmp6 from ($ext_if) to any keep state # For fwjail's own pings

# Allow traffic from internal jails ($internal_net_v4 and $internal_net_v6)
# to pass IN through the internal interface ($int_if).
# These rules create states, allowing return traffic automatically.
# IPv4 from internal jails:
pass in log quick on $int_if inet proto tcp from $internal_net_v4 to any port $allowed_tcp_ports_list keep state
pass in log quick on $int_if inet proto udp from $internal_net_v4 to any port $allowed_udp_ports_list keep state
pass in log quick on $int_if inet proto icmp from $internal_net_v4 to any icmp-type echoreq keep state # Allow pings from internal

# IPv6 from internal jails:
pass in log quick on $int_if inet6 proto tcp from $internal_net_v6 to any port $allowed_tcp_ports_list keep state
pass in log quick on $int_if inet6 proto udp from $internal_net_v6 to any port $allowed_udp_ports_list keep state
pass in log quick on $int_if inet6 proto icmp6 from $internal_net_v6 to any keep state # Allow all ICMPv6 from internal

# Allow NATted traffic (already processed by 'nat on' rules) to pass OUT on the external interface.
# The state established by the 'pass in' rule on $int_if should handle the session.
# These 'pass out' rules ensure the (now NATted) packet is permitted to leave.
# No 'keep state' is needed here, as the state from the 'pass in' rule governs the session.
pass out log quick on $ext_if inet from $internal_net_v4 to any
pass out log quick on $ext_if inet6 from $fw_ext_ipv6_gua to any


# --- Port Forwarding (Redirects / RDR - Examples) ---
# Uncomment and modify these lines to forward external ports to internal jails.
#
# Example: Forward HTTP (port 80) from the firewall's external IP
# to an application jail (e.g., appjail1 at 192.168.100.10).
# rdr pass on $ext_if proto tcp from any to any port 80 -> 192.168.100.10 port 80
#
# If using 'rdr' rules like the one above, you also need corresponding 'pass in' rules on $ext_if
# to allow the (now redirected) traffic to pass *into* the firewall towards the internal host.
# Example: Allow the redirected HTTP traffic to appjail1.
# pass in log quick on $ext_if proto tcp from any to 192.168.100.10 port 80 keep state

III. The Application Jail (appjail1)

This jail will host your application and will have its traffic routed through fwjail.

A. Creating the Application Jail Filesystem

sudo mkdir -p /usr/jails/appjail1
sudo bsdinstall jail /usr/jails/appjail1
# Or, if you have base.txz: sudo tar -C /usr/jails/appjail1 -xvf /path/to/your/base.txz

B. appjail1 Configuration (/etc/jail.conf.d/appjail1.conf)

# /etc/jail.conf.d/appjail1.conf (on the FreeBSD Host)
appjail1 {
  $epair_idx = 2; # Unique index for this jail's epair

  $host_if = "epair${epair_idx}a";
  $jail_if = "epair${epair_idx}b"; # This becomes epair2b inside appjail1's rc.conf

  host.hostname = "appjail1.yourdomain.com"; # Set your app jail's hostname
  path = "/usr/jails/appjail1";
  mount.devfs;
  devfs_ruleset = 4; # Standard jail devfs ruleset (no special PF access needed)
  exec.clean;
  persist;

  vnet; # Enable VNET

  exec.prestart += "ifconfig epair${epair_idx} create up descr jail_appjail1";
  exec.prestart += "ifconfig bridge1 addm ${host_if}"; # Connect to internal bridge1
  vnet.interface = $jail_if; # Assign epair2b to the jail

  exec.start += "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";

  exec.poststop += "ifconfig bridge1 deletem ${host_if}";
  exec.poststop += "ifconfig ${host_if} destroy";
}

C. appjail1 Internal Configuration (/usr/jails/appjail1/etc/rc.conf)

# /usr/jails/appjail1/etc/rc.conf (edit this file on the host)
hostname="appjail1.yourdomain.com" # Match host.hostname from jail.conf

# Network interface (e.g., epair2b, connected to host's bridge1)
# The name 'epair2b' corresponds to $jail_if from appjail1.conf.
ifconfig_epair2b="inet 192.168.100.10/24" # UNIQUE IP for appjail1 on internal network
ifconfig_epair2b_ipv6="inet6 fd00:cafe:beef:100::10/64" # UNIQUE ULA for appjail1

# Default gateway is fwjail's INTERNAL interface IP
defaultrouter="192.168.100.1"
ipv6_defaultrouter="fd00:cafe:beef:100::1"

# Example: Enable a service like SSH or your application
# sshd_enable="YES"
# your_application_enable="YES"

D. DNS Configuration for appjail1

Copy your host’s DNS configuration or create a new one for appjail1. Edit /usr/jails/appjail1/etc/resolv.conf on the host:

# /usr/jails/appjail1/etc/resolv.conf
# Example DNS servers:
nameserver 1.1.1.1   # Cloudflare DNS
nameserver 8.8.8.8   # Google DNS
# Or use your preferred/internal DNS servers

IV. Port Forwarding Example: Exposing a Web Service

Let’s say appjail1 is running a web server on HTTPS (port 443) and you want to make it accessible via fwjail’s external IP address.

A. Concept

Traffic arriving at fwjail’s ext_if on port 443 will be redirected (NATed) to appjail1’s internal IP (192.168.100.10) on port 443.

B. Modifying fwjail’s pf.conf

Edit /usr/jails/fwjail/etc/pf.conf and add or uncomment the following lines in the “Port Forwarding” section:

# --- Port Forwarding (Redirects / RDR - Examples) ---

# Forward HTTPS (port 443) from the firewall's external IP ($ext_if)
# to appjail1 (192.168.100.10) on port 443.
# 'rdr' rules are processed before filter rules for incoming packets.
# 'pass' on an rdr rule creates a state and allows the initial packet.
rdr pass on $ext_if proto tcp from any to ($ext_if) port 443 -> 192.168.100.10 port 443

# If your rdr rule doesn't include 'pass' (e.g., 'rdr on $ext_if ...'), or for more explicit control,
# you also need a corresponding 'pass in' rule on $ext_if
# to allow the (now redirected) traffic to pass *into* the firewall towards the internal host.
# The 'rdr pass' above should handle this, but an explicit pass rule is also common and provides logging:
# pass in log quick on $ext_if proto tcp from any to 192.168.100.10 port 443 keep state

The ($ext_if) in the rdr rule ensures it only applies to traffic destined for an IP address on the external interface.

After adding/modifying these rules, reload pf.conf in fwjail:

# Inside fwjail
sudo pfctl -f /etc/pf.conf

V. Starting and Testing the Setup

  1. Start the Jails: It’s generally good practice to start fwjail first, then appjail1. 
    sudo service jail start fwjail
sudo service jail start appjail1
  2. Verify fwjail Operation:
    • Inside fwjail (sudo jexec fwjail csh):
      • sysctl net.inet.ip.forwarding net.inet6.ip6.forwarding (should both be 1)
      • pfctl -s rules
      • pfctl -s nat
      • pfctl -s info
      • Test fwjail’s own external connectivity: ping 8.8.8.8, ping6 google.com
  3. Test appjail1 Connectivity:
    • Inside appjail1 (sudo jexec appjail1 csh):
      • Ping fwjail’s internal IP: ping 192.168.100.1, ping6 fd00:cafe:beef:100::1
      • Ping an external IP: ping 8.8.8.8
      • Ping an external IPv6 host: ping6 google.com
      • Test DNS: host google.com
  4. Test Port Forwarding: From an external machine (outside your network), try accessing https://<fwjail_external_IP>.

Conclusion

You now have a robust VNET jail setup with a dedicated firewall jail managing traffic for your application jails. This provides excellent isolation and granular control over network access. From here, you can further customize your PF rules, add more application jails to the internal network, and explore more advanced PF features like traffic shaping or VPN integration.

Remember that firewall configuration is critical for security. Always test your rules thoroughly and consult the OpenBSD PF User’s Guide (which is the authoritative source for PF, and FreeBSD’s PF is closely based on it) for detailed information.

Categories:

Updated:

You may also enjoy

Welcome to my personal blog site

less than 1 minute read

Welcome to my personal blog website. This is still highly in construction and my attempt to write down notes, interesting technical findings and experiments,...

Welcome to Jekyll!

less than 1 minute read

You’ll find this post in your _posts directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different wa...