Wednesday July 14, 2021
Exploring the System
So, now, I am interested in how my system works. What are the load of files beyond my User file? I am using them all the time but I for the most part it is a black box to me. But, if I am not understanding where my packages are going when they install or where various versions of Python are located, I will be a disaster when I move over to using someone else’s system.
For example PATH is:
usr/local/bin:usr/bin:bin
If you don’t know what that means, they are a list of directories separated by a : colon. When you write a command such as python or pip, your OS searches through the list of directories in reading order. So, the last directory searched is simply your bin folder.
I have also tried to find all the version of Python in my system. These are the locations that I can find:
usr/local/bin/python3.9
usr/local/lib/python3.7
usr/local/lib/python3.8
usr/local/lib/python3.9
Library/Python2.7
My Initial Impressions
I concluded that what is in usr/local/bin might be something like a link or pointer that is using the indicated python version contained in usr/local/lib. This is must be the python that activates whenever I write python3. I also imagine that if the version doesn’t exist in lib, then it can’t be used in bin (ihdk). The 2.7 version is in the capitalized directories which must be part of the native OS and I should never change anything in these directories. I understand that because macOS terminal is not case sensitive I conclude that these capitalized directories are used to indicate to the user which directories are the system directories. These must be “special” or “native” to the system and if anything changes here it could break your OS.
We are only looking at lowercase directories. I am first using my intuition about this but, naturally, I want to learn more about the purpose of all these directories. For example, what is the difference between usr/local/bin and usr/local/lib. That question could be simplified to the difference between bin and lib, since I find both in all three directory levels starting at top where the root is(/), moving through to usr , and then to local. That is to say at all levels of the PATH there is a bin and a lib directory.
What More Informed People Have to Say
Here the unix stackexchange gives us a less snarky of an answer than what you might find with the stackoverflow kids, and it’s a nice general explanation. What I could find was that bin stands for “binary” which became somewhat synonymous with “executable”. So, I loosely understand that bin contains executables and that must be why the version of Python in bin is the one that runs when I call python3.
Another distinction is that local is simply a folder to indicate where third party Unix software could be installed. If any, major changes to the system came in like an update then these files would be untouched as not part of the native or else they could be exported to en external device and then reimported (I don’t know if I am using the right words). But simply usr/local/bin is for programs that a normal user may run and usr/local is for use by the system administrator when installing software locally.
If you want a more thorough and technical description check out the Filesystem Hierarchy Standard for some bedtime reading. Be careful though, it did cause me to dream about a labyrinth of systems architecture.
Now for what I needed to do from the start
I am still on pause about fixing my problem. It is painful to know that I have so much knowledge and expertise to catch up on. I am learning as fast as I can but as deliberate and detailed as necessary for retention. I don’t know enough about the inner workings of my computer I’ll admit and I’ll take this detour calmly and observe what I can. I’ll put all this together this week and be back on track by next week. But the experience is a little like Alice down a hole in the garden of forking paths and the rabbit running out of time….and there’s always another hole to fall into. If you don’t leave a trail of bread crumbs you’ll never get out.
I understand that I should be using a combination of pyenv and pyenve-virtualenv and possibly pipenv. But there are still some treacherous waters to navigate before we arrive.
pyenv creates what is called a shim which is a directory that is inserted at the front of your PATH.
(pyenv root)/shims:usr/local/bin:usr/bin:bin
These shims are lightweight executables that pass your command along to pyenv.
So, when you pass a Python command, your OS will:
- search PATH for an executable file with the command name you want
- find the pyenv
shimwith that command name at the beginning of the PATH - run the
shimwith that command name and passes the command to pyenv
Okay, let me take a stab at translating this into an analogy. If a friend does a favor for another friend and they arrive to box office early to buy a ticket for a movie. This friend (pyenv shim) goes to stand in in the right line for the right movie (which would be the command PATH) and then the other friend (the chosen version of python) arrives later without interrupting any normal processes to buy the ticket. I’ll accept the faults in the analogy in order to have something to hold onto as an approximation to be revised later. Maybe someone could help revise it for me.
I think this is a good place to stop. Let’s continue this tomorrow….if I keep writing I won’t get any further and we have pyenv-virtualenv, virtualenv-wrapper, and pipenv with Pifile.lock (which sounds similar to requirements.txt, but ihdk).