16 Days of Data / Round 2

Tuesday July 13, 2021


Python environment webcomic by xkcd

Python environment webcomic by xkcd

After the dust settled, I found that not managing my environments well was the true culprit of this rude awakening. My packages all disappeared because I had continued to install packages directly on my system. Since I was using pip3, they were going to my default python3 package directory which was set for version 3.8. While I tried to install a package with Homebrew, I was prompted to update Homebrew. This automatically installed Python 3.9 and that was set as my default python3 version. The package folder for the new version is empty, sans the basic 4 or 5 packages.

So, the solution is easy. I just need to change the direction of my default python3….right?

It’s complicated

The problem is changing the default python3 would only be a band-aid and could potentially cause some unwanted problems. We ultimately want to future proof this. So, we need to start to get a clear picture of how to manage the packages, different versions of the packages and different versions of python.

The real solution is using a combination of pyenv and venv. Pyenv will help us control the version of Python we are using and gives us the possibility of using any version we need. And venv (short for virtual environment, sometimes called virtualenv from what I can tell) is a way to setup isolated environments where each environment contains it’s own packages and versions of those packages in any given directory where the environment is initialized.

So each project directory can have its own environment with its own packages installed. That means we can have different versions of each package in different directories. From what I understand these packages are not installed onto the the system but into the environment. I am still unclear how this relates to –user installations though. I imagine that I should always be treating my own system as if it’s not my own, but as a user of that system. But the fact is I will change the way I do everything from now on.


I will be monitoring this situation as everything develops. Because this is a daily blog, I will not have a complete and thorough rundown of all details. I will leave a list of resources that I am taking notes from to identify exactly what steps to take before I take them.

This article by Shinichi Okada on towards data science reflects my general situation. He has detailed the steps that he followed and that will be the likely path that I follow.

Real Python is always there for you when you need them in this article about pyenv.

This article by Matthew Broberg in opensource.com also gives me a nice road map to compare with.

GiteHub repos:

Even if this has not happened to you yet, you should inform yourself and implement it, sooner than later.




Leave a Reply

Your email address will not be published. Required fields are marked *