Micro services inside FreeBSD using Jails, NGINX and PF

Introduction

My website opens3.net is built using Kotlin (Ktor), NGINX running on a FreeBSD droplet.

The micro services now consist of the following (all written in Kotlin and Perl):

1. Content Management System
2. Weather warning SMS bot (sends weather alerts it parses from BOM)
3. Weather warning back-end
4. Log file processor and IP to location converter for my Ad Blocking Android app

Most of these projects are Open Source.

The configuration files for this write up are available here.

Overview

My FreeBSD droplet runs on Digital Ocean and has a floating IP address. I have configured two jails. One to run the micro services and another to run the database. At a later date I can export the complete jail and run it on a new server or run it at home without having to mess around with importing and exporting databases.

Configuring Jails

The first step is to configure the jails. I am using ezjail. Reference documentation is here.

# pkg install ezjail

First we need to configure our rc.conf to enable our cloned interfaces. Add this to rc.conf:

cloned_interfaces="lo1 lo2 lo3 lo4"

These interfaces will be brought up at boot. To bring up without a reboot:

# service netif cloneup

The following commands will configure the jails to match the previous diagram.

# ezjail-admin install
# ezjail-admin create jvm 'lo1|10.200.1.1'
# ezjail-admin create database 'lo2|10.200.2.1'
# ezjail-admin start jvm
# ezjail-admin start database

In order to have internet inside the jails you will need to add a name server to /etc/resolv.conf inside of each jail.

# ezjail-admin console jvm
# echo "nameserver 8.8.8.8" > /etc/resolv.conf

Then we need to configure NAT in pf by editing /etc/pf.conf:

# Interfaces
wan="vtnet0"
local="vtnet1"
jail1="lo1"
jail2="lo2"
jail3="lo3"

# Options
set block-policy return

set skip on lo0
set skip on $local
scrub in all

# NAT
nat on $wan from ($jail1:network) to any -> ($wan:0)
nat on $wan from ($jail2:network) to any -> ($wan:0)
nat on $wan from ($jail3:network) to any -> ($wan:0)


# default block
block in all

# out is ok
pass in log quick on { $wan } proto { udp tcp } from any to any port {  22 53 80 443 2211 1194 5000 5030}
# Our microservice ports 
pass in log quick on { $jail1 } proto { udp tcp } from any to any port {  3000 8080 7890 }
# MySQL and Redis ports
pass in log quick on { $jail2 } proto { udp tcp } from any to any port {  3306 6379}
pass in log quick on { $jail3 } proto { udp tcp } from any to any port {  3306 8080 }
pass out log quick all keep state

# icmp all good
pass out log inet proto icmp from any to any keep state
pass in log quick inet proto icmp from any to any keep state

In order for NAT to work we need to add the following to our rc.conf:

# Enable Packet Filter
pf_enable="YES"

# Enable Packet Forwarding
gateway_enable="YES"

Start packet forwarding (reboot or enter the following commands):

# sysctl net.inet.ip.forwarding=1
# service start pf

Internet should now work inside both of the jails.

Configuring NGINX

My NGINX configuration is rather complicated as I have the following micro services running:

1. Content Management System
2. WeatherBot Administration Backend
3. GoAccess NGINX web log analyzer

Edit the /usr/local/etc/nginx/nginx.conf file.

worker_processes  1;

events {
	worker_connections  1024;
}

http {
	include       mime.types;
	default_type  application/octet-stream;
	
	# Log Format for GoAccess		
	log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_response_time"';
	# Store log file in JVM jail
        access_log  /usr/jails/jvm/var/log/http-access.log  main;

	sendfile        on;
	keepalive_timeout  65;
	gzip  on;
	
	# OpenS3.net main website
	upstream myapp {
		server 10.200.1.1:8080;
	}
	# WeatherBot admin backend
	upstream noti {
		server 10.200.1.1:3000;
	}
	# HTTP website. Redirects to HTTPS
	server {
		listen 80;
		server_name localhost;
		location / {
			proxy_pass http://myapp;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Proto $scheme;
		}
		return 301 https://$host$request_uri;

	}
	# HTTPS website. Certificate provided by lets encrypt certbot.
	server {
		listen       443 ssl http2;
		server_name  localhost;

		ssl on;
		ssl_certificate    /usr/local/etc/letsencrypt/live/opens3.net/fullchain.pem; 
		ssl_certificate_key /usr/local/etc/letsencrypt/live/opens3.net/privkey.pem; 
		location / {
			proxy_pass http://myapp;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Proto $scheme;
		}
		# Catch the /ws for GoAccess and route to GoAccess daemon.
		location /ws {
			proxy_pass http://10.200.1.1:7890;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "Upgrade";
		}
	}
	# WeatherBot Admin Backend
	server {
		listen       5000 ssl http2;
		server_name  localhost;

		ssl on;
		ssl_certificate    /usr/local/etc/letsencrypt/live/opens3.net/fullchain.pem; 
		ssl_certificate_key /usr/local/etc/letsencrypt/live/opens3.net/privkey.pem; 

		location / {
			proxy_pass http://noti;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Proto $scheme;
		}
	}
}

Running the GoAccess Web Log Analyzer

To run the GoAccess web log analyzer add the following to the /home/nobody/.goaccessrc file:

time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^"

Run GoAccess using the following command to run it as the nobody user.

# su -m nobody -c 'goaccess -p /home/nobody/.goaccessrc /var/log/http-access.log -o /home/nobody/www/report.html --real-time-html --ws-url=wss://opens3.net:443/ws --daemonize'

Final thoughts

Running processes in a jail gives us security, control and flexibilty. If you have multiple CPUs you can dedicate a CPU to a jail. You can also limit memory usage.

You can migrate jails to another machine, or backup the jail to save its state.

Using top you can view the processes running is a specific jail:

GoAccess is a great little tool, to visit my websites state you can view it here (only works in Google Chrome).