How to Work Around Python PEP 668
PEP 668 introduced breaking changes to pip.
Pip can no longer be used as before to install Python applications and dependencies from PyPi.
They've introduced an "externally-managed-environment" lock which prevents pip from performing it's main functionality. When you try to install something, you will now get an error like this:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
This change was made to ensure stability for the system Python packages. If a user installs their own version of a dependency, it could conflict with what was installed for the system and create incompatibility issues. Their way to prevent this is to prevent the user from installing anything at all.
What to do instead
Alternative methods must now be used to perform the same tasks as before.
The functionality of pip is scattered across multiple tools.
If anyone on the Python team dislikes the suggestions here, please update the official docs to provide better solutions.
Install a Python program
Check first if the program you want is available through your system's package manager. Install it through there if the version is up-to-date enough.
If not, you may want to install the PyPi version instead. Install pipx and use it in place of pip to install it.
pipx will automatically handle creating virtual environments for the installed Python programs. You'll be able to use it from the command line like normal, but it will be isolated. You cannot use things installed from pipx as dependencies from local scripts.
Install dependencies for a project
You will need to create a virtual environment for your projects now in order to install and use dependencies.
To do this, create a directory for your project. Enter it and enter the command:
python3 -m venv venv
This will create a virtual environment for the project in the directory 'venv'.
To activate the virtual environment, use the activator script included in the venv directory. There are different scripts depending on which shell you are using. For bash or zsh, use:
source ./venv/bin/activate
You will be able to tell it is active because the terminal prompt will now have a '(venv)' before it.
While the virtual environment is active, it will source it's own versions of pip and the python interpreter. Now if you try to install using pip, you should be allowed, and it will be isolated to the project directory.
To leave the virtual environment, use the command 'deactivate'.
Instead of entering the virtual environment, you can interact with it directly by specifying the path to pip or python inside 'venv'.
./venv/bin/pip3 install numpy ./venv/bin/python3 ./src/__main__.py
Run a project with dependencies as if it was a native script
Since it's not possible to install dependencies globally or to user space anymore, we can't run our script with the native 'python' command. We have to use a workaround to ensure we are using the 'python' that's part of the correct environment for running the script.
Create a virtual environment and build your project as described in the section above ("Install dependencies for a project").
Next, create a shell wrapper for running the script, using a simple Bash script. I suggest giving this script the same name as your project, such as 'project_name.sh'. Whatever you name it here will be the command used to run the Python script.
Add in the hard path to that project's Python interpreter, with the hard path to the main script as it's argument.
#!/bin/sh "/home/username/.local/scripts/example_project/venv/bin/python3" "/home/username/.local/scripts/example_project/src/__main__.py" "$@"
Replace the paths here with the paths for your project as appropriate. If your directory has spaces in the path, remember to wrap them in quotes or escape them.
If you placed the virtual environment in the same directory as your project, you can shorten the wrapper a bit to improve readability.
#!/bin/sh
project_dir="/home/username/.local/scripts/example_project"
"${project_dir}/venv/bin/python3" "${project_dir}/__main__.py" "$@"
Give the script executable permission. Now when the script is run, it will run the chosen virtual environment version of Python to run the script.
Place the shell wrapper somewhere on your system $PATH and you will now be able to run the program directly with 'project_name.sh'. If you want, you can remove the '.sh' extension to make it run with simply 'project_name'.
Distributing a Python project that uses external dependencies
It would be tempting to share the 'venv' directory along with the application to ensure that all the dependencies are there. Unfortunately, this doesn't work because there are different versions of those dependencies for each platform. They are not universal. The person running the application must create their own virtual environment.
To make this simple, you should provide a 'requirements.txt' file with the project. This is a file that lists all the dependencies and their version.
To create a 'requirements.txt' file, create your project directory as before using its own virtual environment. Keep this environment clean, only installing what is necessary for the program to run.
When ready, generate the file using this command:
python3 -m pip freeze > requirements.txt
Include this text file with the distribution. When someone receives it, they can create a virtual environment for themselves. They then run this command to automatically install all the dependencies to it:
pip3 install -r requirements.txt
Now their virtual environment has all the dependencies as appropriate for their system and can run it.
Publishing a Python project, such as to PyPi, is a more complicated process beyond the scope of this tutorial. But if you do this, someone will be able to install your program using pipx.
https://realpython.com/pypi-publish-python-package/
Distributing a Python project as a ZipApp with dependencies included
Another workaround is to directly include the dependencies with your project. This avoid the need for the receiver to need to set up their own virtual environment, because everything is self contained. The downside is that only certain kinds of files can be included this way.
This requires completely restructuring your project. Make sure to read the source code licenses of anything you include to ensure you have permission to redistribute it in this way.
Create a directory for your project. Add a '__main__.py' file to the root. This will be the entry point for your project.
Instead of downloading the dependencies through pip, download them from whatever repository they're hosted in and add them directly to your project directory.
Add any dependencies as either single file scripts or sub-directories. When you use 'import', it will check these adjacent locations first. Some dependencies may need to be rewritten to work this way.
For example, say I want to package a script that uses numpy. I'd set up my project like this:
- project_dir/
- numpy/
- all the numpy files
- __main__.py
For details on how the import system works for local resources, read this tutorial from RealPython:
https://realpython.com/python-import/#packages
Now to package it, we will use the built in 'zipapp' module.
Run:
python3 -m zipapp project_dir -p '/usr/bin/env python3' -c -o output_name.pyz
Where 'project_dir' is the directory of your project, and 'output_name.pyz' is the path you want the output file saved to and named.
This outputs a zip file of the project with a shebang concatenated to the front of it. This shebang allows the zip file to be run directly without needing to specify the Python interpreter. (The interpreter still needs to be installed, this just tells the computer automatically where to find it.)
Give it execution permissions and run it directly with './output_name.pyz', or run it using the interpreter with 'python3 output_name.pyz'.
You can now share this file with anyone and let them run it the same way.
Check out this tutorial for more detailed info on building Python ZipApps.
Write scripts for your system only
Create a virtual environment somewhere that's going to be shared. Install any dependencies there.
Then change the shebang of your Python scripts to reference the specific instance of Python in that directory.
#!/home/me/.local/python3/venv/bin/python3
This has the massive downside of being completely non-portable. Anyone you share the script with will have to manually edit the shebang to point to their own virtual environment. There is unfortunately no standard location for a user's virtual environment. It's not possible to use the 'env' command for this.
Also, I haven't checked, but I don't believe shebangs can have spaces in them. If you have a space in your username, you're out of luck.
The upside is that you can have all your personal scripts reference the same virtual environment, which makes updating easier.
Transition to another language
The most labor intensive option. Rewrite your scripts in a language who's environment does not have these sorts of issues.
Some languages have tools for interacting with Python, which could ease the process of transition.
Julia has PyCall to communicating with Python.
Lua – This fork of Lunatic Python opens up two-way communication between Lua and Python.
Breaking the pip lock
It is not recommended to bypass the lock set in place on pip, but it is possible to do so. These will restore functionality to pip as it behaved before.
Use the command option
Use the flag '--break-system-packages' to bypass the lock. When doing this, remember to use the '--user' flag too, to install packages to the user space instead of system space.
pip3 install --user package_name --break-system-packages
Config file
You can set an option to pip's config file to make the '--break-system-packages' option always true, removing the need to include it each time it is run.
Modify '~/.config/pip/pip.conf' and add:
[global] break-system-packages=true
To find other locations that the config file may be located, run the command 'pip3 config -v list'. The output lists all the locations that pip checks to put the file.
If the file doesn't exist in any of the locations, create it. Use a location that's listed as a variant 'user'.
Remove the file lock
The most destructive option is to remove the lock file entirely.
The file is possibly located at '/usr/lib/python3.11/EXTERNALLY-MANAGED'. The location may be different on different systems.
sudo mv /usr/lib/python3.11/EXTERNALLY-MANAGED /usr/lib/python3.11/EXTERNALLY-MANAGED.bck
pip checks for the presence of this file to decide if it should work or not. The file itself is an INI file that contains the error message to be displayed when it is found. If pip doesn't find this file, it will work as normal.
/knowledge/