Thoughts on Python PEP 668
PEP 668 is a change that breaks pip by allowing setting an "externally managed environment" option that prevents pip from being used by the user.
Literally, the entire point of this is to have an option that stops pip, the package manager, from providing package manager functionality.
I managed to avoid this for a while because I was still using an older version of Debian with Python 3.8. But I recently updated, and now I have to deal with it.
I don't like this change. Copying Ruby is not the way to go. I understand why it was necessary, but it's hugely breaking and left gaps in usability.
Mainly, there's now no simple way to write a quick script for yourself and pull in a dependency as needed. Now you have to create a virtual environment for it, which requires setting up a project directory, which requires setting up project organization, and then if you want to actually use it, you have enter in a long hard path to the correct environment, or write a shell wrapper for it.
The amount of overhead to do something that used to be simple is too high. The simplicity and ease-of-use Python had was it's main appeal. That's now all gone, unless you never use dependencies. Never again can you whip up a quick script to do something once and then never see it again. The only path now is to set up a full on development environment. I suppose forcing developers to do something in a specific way isn't off brand for Python.
The documentation is lagging far behind. They don't offer complete solutions for everything that used to be possible, if they do at all. It disproportionately affects new developers more than experienced ones, who will have to learn more to reach ground level, hurting Python's reputation as an introductory teaching tool.
I've spent months (on and off) trying to find various workarounds for the issues this change causes. It's only today I think I finally found a way to plug the last gap. There's nothing out there about how to write a simple script with dependencies now.
I would have preferred that the Python package manager instead be tweaked to act as, well, a package manager. Track the dependencies of installed applications. Track their versions. Allow multiple versions of a library to be installed at once, when necessary. As a bonus, 'pip update' could actually be a reality instead of the ridiculousness of:
pip3 list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip3 install -U'
Projects could specify what range of versions they accept inside their '__init__.py'. Then when it runs, it fetches the appropriate version. It doesn't matter where it's installed. Make that necessary for publishing to PyPi like 'requirements.txt' is. If they don't specify, use whatever the most recent version is like it currently does.
Easier said that done, I'm sure, but putting everything inside virtual environments feels like a step backwards. My favorite thing about package managers is how it's possible to share dependencies between multiple programs to reduce the total space on disc. Now we have to throw up a bunch of disparaging isolated environments and re-install the same things multiple times.
The implementation of virtual environment feels janky, with it's super state-y 'source' technique. Reading through the 'activate' script doesn't give me much confidence in it either, when there's comments explaining how it could possibly fail. Think I'll just restart the terminal to start fresh instead, thanks.
I'd even be okay with the proposed idea of isolating system space and user space ("System Python"). pip already distinguishes between 'global' (system), 'user', and 'site' locations. The solution they chose only works for the global and site spaces; not user.
Yeah, I read about how Fedora tried and failed at it. They were at a disadvantage because the upstream Python project didn't support it. If they did support it, it would go a lot more smoothly. Just because Fedora failed to do it doesn't mean the idea itself is bad. (Fedora has never been a prime example of stability and good design. Debian's version seems to have had no issues, as long as pip wasn't run using sudo. So, block pip from being run with sudo.)
The Fedora thread discussing why they are reverting away from trying to make System Python work explains that the issue has to do with a deeper design issue of Python. Python could introduce standard locations for this technique, which in turn would require much less workaround. To reiterate: the issue was not the technique itself; it was the lack of upstream support for the technique.
In the end, however, the change was not successful. To make the Platform-Python stack completely separate, its files had to be forced to non-standard locations which, among other things, created enormous problems when trying to build C extensions for it. On top of this, Platform turned out to house many more Python tools and building all the Python dependencies for them was just too costly in this manner.
The PEP's position on this stance is confusing when they spent a significant portion of words under "Recommendations for distros: Create separate distro and local directories" saying to do what is basically a partial application of this idea. The '/usr/local/lib/' path isn't going to ever get used when there's no way to install anything to it now.
By the current recommendations, system packages should go in a virtual environment to isolate them, which no one does. Why not? My assumption would be that it's challenging to regularly access Python scripts inside a virtual environment. So they push this issue off onto users instead.
Let's face it. pip was never well designed in the first place. It needs a change. A big change. They went with this small change because it was the minimal development effort. But I really would rather we take a big step that fixes major underlying issues, rather than a small patch that's easy for devs, breaking for the users. I do not agree that a change that completely changes the fundamental basics of how one interacts with the ecosystem is somehow a minimal impact change.
The loss of the user space is saddening. This was a space someone could quickly write a small script (aka "programming in the small") in a more lax manner. This is necessary for Python to work as a glue language. Virtual environments are good for the site level, but not user level. Now I need another language just to run Python like I could before, so why don't I just write in that first language?
[Wikipedia] Programming in the large and programming in the small
Now everything Python must be either system (set by whatever organization you're under) or application level (in a virtual environment). No more room for the person. It's the old "we're doing it for your own good".
The '--user' flag in pip should have been the default from the start. Now it's a useless option. You can't use it in the default environment, and you can't use it in virtual environments (it breaks things there).
Keeping all of these different virtual environments up-to-date independently is going to be such a pain. It would be super convenient if there was a tool we could use to update everything we had installed all at once.
The decision has already been made, so we're going to be stuck with it for a while. I wrote about my current workarounds, for those curious.
I'm seriously considering just disabling the file lock so I can have user space back again. But having that option seems like the first step of phasing it out forever, so I'm going to try finding a way to make it work before it's forced entirely.
Alternatively, I'll consider using some other languages that can run natively without a wrapper.
Perl – What was popular before Python. Powerful cross-platform scripting.
Lua – Another popular language that's easier to learn, more standard, and smaller than Python.
Janet – A tiny, lightweight LISP style language that feels more like Python.
/gemlog/