f451 Labs piDEV - A Local DEV server

Overview

The main objective of this project is to run a development web server on a Raspberry Pi device for several website projects. Most of my websites are created with the static site generator Hugo, and the actual production versions of the sites are hosted on DigitalOcean. However, the development versions run locally on my laptop or this Raspberry Pi device.

Why do I have 2 different development environments? Mostly because I had this extra Raspberry Pi lying around and didn’t know what else to do with it. Seriously, though, I often work on things I want to show someone “outside” my (home) network. Then there are times when the version on my laptop just isn’t stable enough, or I want someone to review the latest content updates before it’s pushed to production, and so on.

In short, it’s great to have a place to put stuff for review before things go out to the world 😊

Device specifications & configuration

General specifications

I decided to use this (fairly) powerful Raspberry Pi so that I can host various types of projects and websites without worrying much about performance.

• Device: RP4 - Model B Rev 1.4 / 8GB - Bullseye (Debian 11) / 64bit
• Network: wlan (wifi)
• Display & keyboard: shared resources via KVM
• Other hardware: Argon Fan HAT, Argon Neo case
• Remote access: SSH, VNC

Main tools & applications

Most software on this device focuses on serving websites. But I’ve also added a few tools to make life easier when I need to debug stuff after things go sideways.

• Nginx — web server
• Ngrok — safely manage tunnels to/from outside
• NetworkManager — my default tool for managing network settings on my RPIs
• UFW (firewall) — my default tool for managing firewall settings on my RPIs
• htop — my default process viewer/manager on my RPIs
• Misc. tools — e.g. PIPX, git, etc.
• SQLite — file-based database used for some projects

As a side note, most of my devices are configured in similar ways with (mostly) the same tools. And while some devices have GUIs, most do not, and I usually access all devices from my laptop via SSH.

Common workflows

Given its role as a development server — yes, I know this isn’t really a “server” in the traditional (DevOps) meaning, but this little guy can definitely pull his own weight! — this Raspberry Pi device is part of various workflows. However, most, if not all, workflows start on a different system.

Create and publish content for static sites

Most of my sites are built with Hugo, which means that the content consists mainly of markdown files and various types of images or media files as needed. The most common tasks are:

1. Manage notes and ideas for blog posts in Obsidian — this tool is my “2nd brain” and I have it running on my laptop, phone, and tablet. Obsidian use markdown files as well, and I often start writing drafts for new content in Obsidian.
2. All website files are managed with ‘git’ and stored in a GitHub repo — this is the default for all my projects.
3. Create a new git branch for the new content files — this process requires a few extra steps, but it’s well worth it. Having multiple branches also allows me to work on several things in parallel and only push out stuff to the world that is ready for public consumption. I usually create a new branch before I create any new markdown files. Also, note that content in Hugo is often organized in folders, which lets us keep media and other resource files close to the markdown file. Of course, that means that we’re now potentially managing several items for each blog post, and I find that having a new git branch ready usually helps control the chaos.
4. Write and tweak content in Grammarly — these days I run almost all content through Grammarly to ensure that my sites have as few mistakes as possible.
5. Finish editing markdown files on my laptop in VS Code — this has become my default IDE, and I also use it for editing markdown files. It has an okay-ish “markdown viewer,” and I usually don’t feel the need to use any of my other editors. Another upside with VS Code is that I can easily access the source files for the entire site and tweak templates, modules, config files, or anything else as needed.
6. Continuously edit and review content and media locally using the Hugo built-in server — I work with a double-monitor setup and have VS Code in full-screen mode on one screen, two terminal windows and a web browser open on the second screen. One terminal runs the built-in Hugo web server via the hugo serve command, and it automatically refreshes when I save a file to the project folder. This lets me see how the new content would look on the site in a browser, and, of course, with the browser development tools, I can also simulate what new content would look like on other devices and screen resolutions.
7. Push clean content to Raspberry Pi using ‘rsync’ — from time to time, I push review-ready content to this Raspberry Pi. This is usually the case when I work on longer posts or pages, which can take some time and often benefit from being reviewed by someone other than me. Of course, Hugo is a static site generator, meaning we first need to “generate” the site files before we can push them to the appropriate web folder to be served by Nginx. I have created a small bash script that handles this, and you can read more about this process below in the “Publishing Hugo site content to this DEV server” section.
8. Merge content branch to ‘main’ when everything is ready to go live — normally, I perform this process on GitHub. I create a “merge request,” add some notes, and then “approve” the request. And yes, it may seem a bit silly to go through all these steps as a solo developer. But I find that I have a bit more control over the process this way. Furthermore, as soon as the “merge request” is approved, the ‘main’ branch gets updated, and a GitHub “action” automatically deploys the new content to production on DigitalOcean.

Build and test Python web applications and services

At this point, I’m (slowly) working on 2 projects in this category, and my goal for both projects is to learn (new to me) frameworks. The first project is based on Django, and the second is based on the LiteStar framework.

The nature of these projects is obviously very different from Hugo-based websites. So to keep the code on this Raspberry Pi in sync with the code on my laptop, I used git to clone the relevant GitHub repos to the device. I’ll then use the git pull command whenever I want to update the code on the device.

There may be ways to further streamline or automate this, but I’m not sure it’s worth the effort. After all, typing these git commands is not very taxing 🤓

Build and test Python CLI applications

This category of projects is also fairly straightforward. I try to make most, if not all, of my CLI applications installable via the pip command. The key here is to use the pipx command to ensure that whatever application I install does not “pollute” the Python environment on this Raspberry Pi device.

Now, one additional wrinkle is that my applications and packages are not hosted on PyPi. Instead, I install them directly from my GitHub repo as follows:

$ pipx install git+ssh://<username>@github.com/<path-to-my-app.git>

This method allows me to install Python applications quickly, and I can run each CLI application by simply typing my_app_name in the terminal.

The pipx install command is very flexible, and several switches are available. For example, the --python switch allows us to define which version of Python (v3.6+) to use for the installation, which can be very useful when trying different scenarios.

It’s also possible to use the pipx upgrade to upgrade one’s application. However, that assumes that one first has bumped the version number in GitHub. Instead, I rerun the pip install command with the --force switch, which doesn’t care whether the version has been bumped.

Finally, I do not install my applications in “editable” mode (using the -e switch) on this Raspberry Pi as I do not edit code directly on this device.

Publishing Hugo site content to this DEV server

As mentioned above, I use Nginx to serve up the websites on this device. There are, of course, several alternatives, including Apache, Lighttd, and Caddy. However, I prefer to use Nginx as it’s (relatively) easy to configure to host multiple websites and services.

My basic folder layout for the web content on my Raspberry Pi device is as follows:

  /var
    |
    +--/www
    |    |
    |    +--/website_one     # web root for 1st website
    |    |
    |    +--/website_two     # web root for 2nd website
    |    |
   ...  ... 

The Nginx config file for each enabled website points to the web root, lists the port(s) to listen on, and defines other relevant attributes. And since I host several sites and services on a single device, I have to use a different port for each. For example, the first website may use port 8080, the second may use 8081, and so on.

The URL for each site would then be as follows:

http://<IP of device>:<port #>

It’s also possible to enable SSL, and since this is a locally hosted development server, the easiest way is to use self-signed SSL certificates. One can also use the excellent “Let’s Encrypt” service.

There are several great posts on the interwebs on how to set up Nginx on a Raspberry Pi, assign a static IP address, configure the firewall, and more.

To make life a bit easier, I have also installed the SSH key from my laptop as an authorized key on this Raspberry Pi (using the ssh-copy-id command). That means I can use SSH without entering a password when accessing this device. But more importantly, I can also copy content from my laptop to this device using the rsync command without having to enter a password 😎

I have created a simple “deploy” script based on the suggestions from the Hugo documentation:

#!/bin/sh
# File: deploy2rpi
# -- Set main vars
USER=<user name on device>
HOST=<device host name>
DIR=/var/www/<web root for website>   # target dir on remote server/device

# Generate site and transfer files to target folder on
# remote server/device. Also delete any files that do not
# exist in 'public' folder on local (source) system.
hugo -e rpi -DEF && rsync -avzh --delete -e ssh public/ ${USER}@${HOST}:${DIR}
exit 0

The above script does 2 things:

1. Run the hugo command to generate all files for the website
   • the -e flag tells Hugo to use the config file for a particular environment. In this case, the environment is called ‘rpi’, and Hugo will look for config.toml and other files in the rpi folder.
   • the -DEF flags tell Hugo to include content marked draft or has future or expired publishing dates. I use these flags to ensure all content is compiled for transfer to the DEV server.
2. Run the rsync command to copy all files from the local public Hugo folder to the web root on the Raspberry Pi device using SSH.
   • note that the --delete flag will instruct rsync to delete any files from the target folder that do not exist in the source folder.

Now, why did I not just clone the repo to this Raspberry Pi and run the built-in Hugo server? The main reason is that I’d only be able to serve up one website at a time. With Nginx, on the other hand, I can serve up any number of sites and services at the same time.

Summary

Getting this little guy up and running as a development server has been an interesting mini-project. It took a bit more thinking and futzing around than I had thought, and I went down a few different routes before I settled on the current configuration. Of course, I also had to reacquaint myself with Nginx config files, remember how to tweak firewalls, etc.

This time, though, I took the time to document things better in my always-present Obsidian “second brain” so that It’ll be easier to keep this device maintained and configured properly.

All in all, having this device as a place to put stuff for review before things go out to the world is really handy 😊