OPNsense

February 1st, 2021

I've been using pfSense as my firewall/router for several years now.  Overall, I've been very satisfied except for one issue.  I don't believe I ever got failover to work reliably and that is key.  I had written my own software to handle failover years back so it should have been a no brainer to get it to work properly on a dedicated firewall/router package.  For the past couple years, I actually installed an a/b network switch to manually switch between our primary and backup network as needed and utilized a second (backup) server to connect to the AT&T DSL line.  The current state came up in conversation and I decided I should make another attempt at getting it working properly.  During my research process, I stumbled upon OPNsense which is a fork of pfSense and ultimately decided to give it a try.  It turns out that 5+ years of tweaking my router / firewall / DNS / DHCP / etc... adds up to a significant amount of configuration so it took a few days to get everything set up but I finally made the switch and I believe everything (including failover) is working properly now.  Here are a bunch of links that I used to get everything set up (note: some links are for pfSense but are related/applicable).

Note: While apparently it is possible to import some configuration sections from a pfSense export file, I chose to set everything up manually to ensure it was done (theoretically) properly and not take any chances on configuration incompatibility.  In some cases, I did export configs from both systems, extract data from the pfSense backup, clean it up and paste it into the OPNsense config and re-import, but this was mainly for DHCP static address mapping.

Comparison
Thinking about moving from pfSense to OPNsense

Multi-WAN / Failover
How to configure pfSense as multi wan (DUAL WAN) load balance failover router
"Skip rules when gateway is down"-equivalent in OPNSense
(Solved) Multiwan (failover) DNS not working
Default gateway behavior with DNS Resolver in 2.4.4
DNS doesnt work for wAN 2 FAILOVER
DUAL WAN ON PFSENSE 2 - HD Quality
pfSense Part 10: Configure Multi WAN Load Balancing
How to Setup Failover and Load Balancing in PFSense
Pfsense - Multiple Wan link load-balancing
Pfsense - Multiple Wan link Failover Configuration
OPNsense - Multi WAN

Installation
OPNsense Installation [ Step by Step ]

Port Forwarding
NOTE: for port forwarding, set up the port forward and OPNsense will automatically create the linked rule.
Portforwarding in OPNsense
[Tutorial] How I do port forwarding - simple and straightforward
How to Configure Firewall Rules in OPNsense

BIND
OPNsense - BIND Plugin
BIND with DNSBL – more detailed information about DNSBL types?
DNSBL via BIND Plugin

Config Backups
Automatic config backups using os-api-backup
Sample bash script for backups using the API

Miscellaneous
My Successful Migration - pfSense to OPNsense
How to setup AT&T Arris BGW210-700 IP passthrough

Python Selenium for Automation

January 25th, 2021

CherryPy Digest and Basic Authentication Tutorial

January 23rd, 2021

Jim Hoskins posted a tutorial on CherryPy Digest and Basic Authentication and since, his site is no longer available.  The Wayback Machine has a capture of it but just in case it disappears from there, here is a copy for my own reference.

CherryPy Digest and Basic Authentication Tutorial

Many applications require the use of password based authentication. In many web applications, this is done using cookies/sessions, and the login mechanism is an HTML form that is submitted to the application. If you are trying to build a RESTful web service, or any service where this client may be automated, It becomes useful to utilize HTTP authentication.

It may not be immediately obvious from the CherryPy documentation, but CherryPy includes all the tools needed to implement HTTP authentication, both Digest and Basic. You can see what the CherryPy website has documented about authentication at the CherryPy Builtin Tools Page. The best documentation for this is available in the book: CherryPy Essentials: Rapid Python Web Application Development

Basic vs Digest

HTTP supports two types of authentication, basic and digest. To a user accessing a protected page through a web browser, both types look the same. The browser will prompt the user for a username and password. The difference is how the client (browser) sends the password to the server.

With basic authentication, the client sends the password in clear-text, so any malicious attacker who can see the request can see the password. With digest authentication, the client does not send the password, but rather a digest based on the password and other factors. The server will also compute the digest using the known correct password. If the digest sent by the client matches the one calculated by the server, authentication succeeds.

The catch with digest authentication is the users’ passwords must be stored stored in clear-text on the server** in order to calculate the digest. On the flipside, basic authentication must send the password in clear-text through the network. If you are going to use basic authentication, it is recommended you use SSL to prevent the password from being sniffed.

** This is not entirely true, The digest algorithm allows you to store a partially digested password on the server side, however CherryPy has no easy way to specify this.

Digest Authentication

The following code creates a simple application with a public page and a secure page using digest authentication. The server configuration defines that the /secure url is to be protected with digest authentication (via the tools.digest_auth Tool available in CherryPy)

 

import cherrypy
 
class RootServer:
    @cherrypy.expose
    def index(self):
        return """This is a public page!"""
 
class SecureServer:
    @cherrypy.expose
    def index(self):
    return "This is a secure section"
 
if __name__ == '__main__':
    users = {"admin": "secretPassword",
             "editor": "otherPassword"}
 
    conf = {'/secure': {'tools.digest_auth.on': True,
                        'tools.digest_auth.realm': 'Some site',
                        'tools.digest_auth.users': users}}
    root = RootServer()
    root.secure = SecureServer()
    cherrypy.quickstart(root, '/', config=conf)

Run this code and visit http://localhost:8080/ . Now visit http://localhost:8080/secure. When your browser prompts you to do so, enter “admin” and “secretPassword” for the username and password respectively.

The /secure configuration section sets up the digest authentication with the following options:

  • ‘tools.digest_auth.on’: This boolean enables digest authentication for the given section (in this case /secure)
  • ‘tools.digest_auth.realm’: This string defines the realm to supply to the client.
  • ‘tools.digest_auth.users’: This dictionary defines users and passwords ({user: password}). This can also take two other forms, please see below.

There is a bug in the CherryPy Digest Authentication tool that prevents authentication when the request is a post request with a body. See Cherrypy Digest Auth POST Problem (and Solution!)

Basic Authentication

We are going to use the same code to implement basic authentication, but we will change the configuration variables.

import cherrypy
 
class RootServer:
    @cherrypy.expose
    def index(self):
        return """This is a public page!"""
 
class SecureServer:
    @cherrypy.expose
    def index(self):
    return "This is a secure section"
 
if __name__ == '__main__':
    users = {"admin": "secretPassword",
             "editor": "otherPassword"}
 
    def clear_text(passwd):
        return passwd
 
    conf = {'/secure': {'tools.basic_auth.on': True,
                        'tools.basic_auth.realm': 'Some site2',
                        'tools.basic_auth.users': users,
                        'tools.basic_auth.encrypt': clear_text}}
    root = RootServer()
    root.secure = SecureServer()
    cherrypy.quickstart(root, '/', config=conf)

Run this code as before, and supply the same username and password. Can’t tell the difference? Remember, the difference is in how it is transmitted. Here are the options we supplied to enable basic authentication:

  • ‘tools.basic_auth.on’: This boolean enables digest authentication for the given section (in this case /secure)
  • ‘tools.basic_auth.realm’: This string defines the realm to supply to the client.
  • ‘tools.basic_auth.users’: This dictionary defines users and passwords ({user: password}). This can also take two other forms, please see below.
  • ‘tools.basic_auth.encrypt’: This takes a function that encrypts the user-supplied password to match a pre-encrypted password in the user dict. (In our case, the function clear_text we defined just passes the password through, allowing us to store password in clear-text for ease of demonstration). If you do not supply this option, it will be assumed the passwords are stored in MD5.

Defining Users

You can supply either a dict or function to the ‘tools.******_auth.users’ option to define your users. There are three ways to supply users: as a dict ( {user:password} ), as a function that returns a dict ( {user:password} ), or as a function that takes a username as an argument, and returns a password.

Remember that password could mean the clear-text password, or an encrypted password, depending on whether your using basic or digest, and the encryption option specified.

Using a dictionary

Simply supply a dictionary of {username: password} to the users option of the authentication configuration.

users = {'user1': 'password1',
         'user2': 'password2}
 
conf = {'/secure': {'tools.digest_auth.on': True,
                     'tools.digest_auth.realm': 'Some site',
                     'tools.digest_auth.users': users}}

Using a function that returns a dictionary

Supply a function that takes no arguments and returns a dictionary containing all usernames and passwords

def get_users():
    return {'user1': 'password1',
            'user2': 'password2}
 
conf = {'/secure': {'tools.digest_auth.on': True,
                     'tools.digest_auth.realm': 'Some site',
                     'tools.digest_auth.users': get_users}}

Using a Function Which Takes a Username (Most Useful)

Many times, applications have so many users that it does not make sense to pass the list of all users around like the options above. I have not seen this feature documented anywhere, but the code allows it. You can specify a function that takes a username string as an argument, and returns the password for that user. This function could consult a database or other mechanism, and allows your application to add and remove user more easily.

def find_password(username):
    #INSERT PASSWORD LOOKUP LOGIC HERE
    password = db.find_password_for_user(username)
    return password
 
conf = {'/secure': {'tools.digest_auth.on': True,
                     'tools.digest_auth.realm': 'Some site',
                     'tools.digest_auth.users': find_password}}

Retrieving the Username

Once authenticated your application will likely want to know which user is logged in. The username of the user who has successfully authenticated will be available at ‘cherrypy.request.login’. This variable wil be False if authentication failed.

In Closing

It may not be well documented, but the tools for basic and digest authentication are already built in to CherryPy. You just have to know how to use them, and now you do.

If there are any problems with the code or this post in general, leave me a comment and I will make it right.

Deck Project

January 23rd, 2021

When we moved into the house on Appaloosa Drive, there was an existing deck which had some areas that obviously were going to need to be repaired.  The coloring (dark brown) was not particularly appealing to us either.  I had planned on removing the deck boards, fixing the damaged framing, stripping and finally reinstalling the decking boards.  This, however, was not how this project ended up going.  Many of the deck boards were damaged to the point that they would not stand being removed, stripped and reinstalled.  The framing had previously been "repaired" and had sustained significant damage (rot) and there were numerous structural issues with the way the framing had been done.  Ultimately, *everything* needed to come out, brand new framing put in (resolving structural issues) and where possible the old deck boards resurfaced.  I watched *lots* of videos and read many articles researching the appropriate approaches to take.  Below are links to the main references I used.

Series
Home Renovision DIY - How to Build a Floating Deck Step by Step
HouseImprovements - How To Build A Deck
HouseImprovements - DIY Weekend Deck Project
DIY Weekend Deck Project Part 1 - Deck Board Removal
DIY Weekend Deck Project Part 2 - Framing And Leveling
DIY Weekend Deck Project Part 3 - Laying Down Deckboards
DIY Weekend Deck Project Part 4 - Installing Deck Railings
DIY Weekend Deck Project Part 5 - Skirt, Box Steps And Angled Railings
BYOT - How To Build A Deck: Start To Finish
HOW TO BUILD A DECK : START TO FINISH (Part 1 of 2)
HOW TO BUILD A DECK : START TO FINISH (Part 2 of 2)

Framing
Feature Board Framing
DIY Deck Part 7 - Connecting Corner Joists
DIY Deck Part 8 - Preventing Joist Rot
Installing Deck Joists
Joist Layout Ideas for Stronger Decks
Deck Joist Cantilever Rules and Limits
How to Calculate Deck Spans for the Ultimate Outdoor Hangout
How to Add an Angled Corner and Joists to a Deck Frame

Deck Boards and Layout
What is the Proper Deck Board Spacing
How to picture frame your deck

Restoration and Finishing
How To Protect Your Deck | Never Sand Again
How to Restore a 30 Year Old Deck
How-to Refinish Your Deck
The 6 Best Deck Stain Reviews and Ratings
Restore-A-Deck Cleaner System Review
How To: Prepare Your Deck for Stain
Restore-A-Deck
TWP 1500 Series Review
TWP 1500 Stain
TWP 1500 Stain Surface Preparation
Staining A New Deck
Why Sanding a Deck Can be an Issue

General
How to Fix Your Deck | A to Z
Tips for Building a Deck - 10 Tips to Make Your Deck Last Longer
Build a Floating Deck
Star Concrete (concrete disposal)

Python CherryPy

January 23rd, 2021

I've had a number of situations where I would like to set up a python web application.  I've used CherryPy for this once before and just used it again.  Here are a number of CherryPy related links that I expect I'll need to use again if I use CherryPy for another project in the future.

Installation / Home Page
Simple form with CherryPy
CherryPy - Use Of Ajax
File upload with CherryPy
Source code for cherrypy.tutorial.tut09_files (File upload and download)
CherryPy Tutorials
Creating a Simple app with CherryPy and React JS
CherryPy Digest and Basic Authentication Tutorial
Basic Auth for CherryPy webapp
CherryPy - A Minimalist Python Web Framework

 

Here are some links dealing with Session Locking, Performance and Concurrent Requests:

How to make CherryPy serve concurrent requests?
Session Locking and Performance in CherryPy