Publishing a Python OSS library

I decided several months ago to create and publish an open-source software (OSS) Python library. And now, after fighting countless uphill battles, writing and rewriting lots of code, and learning many new skills, I was finally able to publish my first library — f451-comms — to Pypi. And what a journey it was!

The original goals for the project were:

• create a library that I can reuse in my personal (side) projects
• improve my Python coding skills
• learn how to develop and publish an OSS library for the larger Python community
• learn the tooling and processes recommended (or even required) to publish OSS projects And, as a bonus, I’d be able to apply some of the lessons learned in my day job. In short: there would be nothing but upside! 🤓

Of course, as happens so often with passion projects, I completely underestimated the effort required to pull this off. For example, I completely underestimated how long it would take to write the actual code. Although, in my defense, I had already written what I thought would be the bulk of this library while working on another project. How wrong I was! I did not foresee how much new code I would write, and much refactoring would be required to give the whole thing some polish.

I also did not realize how many new tools, processes, and techniques I would need to learn. There were new tools for test automation, for managing Python packages, for managing Python versions and virtual environments, and for code formatting and handling and type checking. And I had to learn how to create complex Python type hints, how to write proper docstrings that tools like Sphinx can parse, and much more. Sure, I had used some of the tools and even experimented with a few techniques before, but not in this coordinated manner.

In this post, I highlight some of the key tools and processes that I’ve learned and why I think they’re helpful. The following three sections — “Writing code,” “Using tools,” and “Publishing the library” — each outline some lessons learned and also list some relevant resources.

Writing code

My coding style improved quite a bit throughout this project. But there was also a lot of refactoring that moved the needle from “this code is ugly but works” to something more polished and better thought through.

Here are a few notes:

• Smaller/shorter functions — Breaking up larger segments made it easier to understand and follow but also easier to reuse and maintain
• more classes and more inheritance — I was already using classes, but with this exercise, I took it to the next level. It required some rethinking and refactoring, but the end result was a simpler and more flexible code base. This effort has changed how I think about my code and the structure of my libraries and applications.
• Type hints — This was a game changer and required a lot of effort. There was a lot to learn, not just in terms of writing code but also around testing, etc. I’m still on the fence regarding type hints, and I did not add type hints to my tests. Going forward, I will add type hints to any OSS libraries, but probably not to code that is not intended for sharing with the world.
• More tests and TDD — This was another big change. I had dabbled with TDD before, and I usually always wrote unit tests. For this project, though, I really had to step up my game. Now, while I definitely understand and value the benefits of TDD, I’m not dogmatic either, and I’ve tried to find a balance that works for me. One thing is certain, though, having decent test coverage has saved my bacon quite a few times, and I will extend this even further.
• More and better documentation — I decided to implement the Google style guide for Python (as best I can). This meant better docstring sections, and I also included better and more code samples. This paid off later when I started to use Sphinx and the ReStructuredText format for user guides.

Using tools

• Pyenv — I switched wholesale over to Pyenv for managing multiple versions of Python and associated virtual environments. In fact, Pyenv is now one of the first tools that I install on new computers where I want to run Python.
• Poetry — This is another core tool, and it helps me manage packages and much more in a given project.
• Pipx — I’m using this when I need to install any standalone Python tool.
• Nox — Helps automate all tests and builds.
• Pytest — My preferred testing framework!
• Sphinx — Helps build user guides written in the ReStructuredText format.
• Mypy — Tests and validates type hints.
• Black and Flake8 — Standard tools for code formatting and style guide enforcement. I may not always agree with every default style choice, but I will assimilate 🤓
• Coverage.py — measures code coverage.
• Hypermodern Python Cookiecutter — a very comprehensive and complete template with lots of tools and (opinionated) guidelines. This template helped me a lot, but it also came with a bit of a learning curve.
• GitHub actions — scripts to help automate many release-related workflows.

Publishing the library

• Writing documentation and publishing to ReadTheDocs — This took a bit of getting used to. First, I switched from writing Markdown to ReStructuredText. Second, building the content with Sphinx and viewing it in my local environment is great. And while it added a lot more flexibility to the documentation, it also added a level of complexity.
• Using pre-commit hooks and Nox for workflow automation — It took some time for me to get all the tools configured properly. But once that was done, running automated pre-commit tests sure saved lots of time and reduced the number of head scratch moments.
• creating releases and publishing to Test Pypi and Pypi — the uploads to these portals were automated with the GitHub actions included with the Hypermodern Python Cookiecutter template. But, I learned the hard way that one must properly set up the Pypi tokens in the library for it to actually work 🤓

General lessons learned

One of the most (hardwon) lessons learned was to invest time and effort in learning as much as possible about relevant tooling and templates. In my case, I had a lot to learn and figure out. Many of the tools were not only new to me but they were also built into various automated processes. And it was not always crystal clear where one would need to update and tweak the settings for each tool so that they would have an effect.

Looking back, though, it was absolutely worth the effort, and I feel that my development environment is now both smarter and much more robust. In fact, I wish I had started with a smaller test project just to learn how to better use the tools and workflows.

All in all, I’m very happy with this project. I learned a lot, and I’m confident that the next project will go much faster. There is always more to learn, but I feel aI have a much better understanding of how things work in the world of Python.