Automatically Installing Python System Dependencies

We’re working on a feature to automatically install system dependencies when your install or add Python packages to your Repl. To make this work, we have a mapping from Python package to what it should install:

{
"pycairo": {"deps": [ "pkgs.pkg-config", "pkgs.cairo" ]},
"moviepy": {"deps": [ "pkgs.ffmpeg", "pkgs.imagemagickBig" ]},
"psycopg2": {"deps": [ "pkgs.postgresql" ], "libdeps": ["pkgs.postgresql"]}
}

For example, if you install pycairo it would install pkg-config and cairo into the replit.nix deps list.

Please help me make this map more complete! Post your python packaging problems below like this:

  1. The name of the Python library that is not working.
  2. A minimal code sample that doesn’t work.
  3. What needs to be added to replit.nix to make it work.

Here’s an example report:

  1. pycairo
import cairo

with cairo.SVGSurface("example.svg", 200, 200) as surface:
  context = cairo.Context(surface)
  x, y, x1, y1 = 0.1, 0.5, 0.4, 0.9
  x2, y2, x3, y3 = 0.6, 0.1, 0.9, 0.5
  context.scale(200, 200)
  context.set_line_width(0.04)
  context.move_to(x, y)
  context.curve_to(x1, y1, x2, y2, x3, y3)
  context.stroke()
  context.set_source_rgba(1, 0.2, 0.2, 0.6)
  context.set_line_width(0.02)
  context.move_to(x, y)
  context.line_to(x1, y1)
  context.move_to(x2, y2)
  context.line_to(x3, y3)
  context.stroke()
  1. Add pkgs.pkg-config and pkgs.cairo to replit.nix deps.

If you don’t know how to fix it yourself, I’d still like to hear about all the Python packages that don’t work for you!

9 Likes

2 python packages I have tried to fix:

  1. PyQt6 (I thought my template worked but the graphics don’t work, can this be fixed or an official template made?)
  2. OpenCV (cv2), there is a older template and I made a newer template too, but it would be easier for people to just install it without a template.

Comment:
I see that the intent is to install the non-python dependencies that are available in nix. When I was fixing the two packages however, I was doing an alternate way:

The way I tried to get these python packages was just with nix, without touching poetry and so not installing locally, because
these python packages are available as “independent” nix packages (see search) in the category of python310Packages (there’s also python311Packages).
The 2 packages above and also the three packages in original post are all in nix as independent packages.

One advantage of these is greatly reduced storage usage if the actual package (excluding dependencies) is very large.
However, these packages aren’t seen by poetry obviously, it’s part of the nix deps.
Most of the time, a python program can find a nix python package (it’s in sys.path) but one time it wasn’t able to.

So maybe installing the dependencies is better, or using the independent nix package is better, I don’t know.

4 Likes

selenium setup takes more than a package in Nix deps. For chrome you need to make sure the versions are aligned and for Firefox it’d take either to modify the code or to add some sort of module loader code to the python wrapper which monkey-patches selenium with the firefox config. See:

(could someone format a selenium request in the proper way which goes with the instructions in the first post? I gtg)

6 Likes

I’ve used a package called magic or libmagicor python-magic from memory (there are a few versions on PYPI) that requires pkgs.files.

2 Likes

Tabula requires Java to be installed or something.

Thank you for all the feedback in this thread. I’m recording it and will circle back and try to fix these dependency issues.

We’re currently releasing a version that has over 700 Python dependency fixes! I generated these fixes in an automated manner using information from Nixpkgs.

To find if you have this version, you can type pid1 --version in the shell:

$ pid1 --version
pid1: 0.0.0-5440be3+nix

You also need to have Explorer enabled on your account settings page.

Please let me know if you notice this helping you or if you have problems because of it and I’ll adjust it as needed!

2 Likes

Hey, there might be a problem happening:

matplotlib, ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

Dang, you’re right, sorry! There is an issue where specifying PYTHON_LD_LIBRARY_PATH in replit.nix causes it to not use the default PYTHON_LD_LIBRARY_PATH values, which will break certain dependencies. I’ve turned off this feature for now, and will work on a fix.

1 Like

8 posts were split to a new topic: Disabling automatic package guessing and installing

The PYTHON_LD_LIBRARY_PATH issue is now fixed, we keep the default entries when new entries are added to replit.nix. I also made some improvements to the mappings so that it wouldn’t, for example, install multiple versions of ffmpeg.

The automatic installer is back out to all Explorers. Please let me know if you notice any issues or have any concerns!

5 Likes

We’re rolling this out to everyone slowly. Please let me know if you notice any issues!

4 Likes

This feature only works for the Packages tool, and import guessing.
The shell commands (pip install, poetry add, upm add) do not install system dependencies.
However, many people use poetry or pip commands instead of the Packages tool because it is more convenient and faster to use.

System dependencies are added when a package is added, but not removed when a package is removed.

The System Dependencies tool also cannot manage shared libraries (libdeps), only deps.

3 Likes

Right, we’ll need to look into if it is possible to hook into various packagers to improve the usefulness of this.

Right, this is tricky to do with the current system since two packages could have added something and someone might have also manually added something.

Right, also. Thankfully there is some good news here. We recently changed how the Nix environment building works to populate the environment variable REPLIT_LD_LIBRARY_PATH with the /lib dirs of all the packages in deps, and we’ve changed the nodejs and python tools to recognize the REPLIT_LD_LIBRARY_PATH, so it is no longer necessary to specify a PYTHON_LD_LIBRARY_PATH in replit.nix. Simply adding the shared libraries to deps is sufficient!

3 Likes

The Python automatic system dependency installer is now rolled out to everyone! Please continue to share cases where Python packages don’t work so we can fix those cases, and let me know if you have any issues with this!

2 Likes

opencv-python (import cv2) does not work (installed with auto guessing)

Traceback (most recent call last):
  File "/home/runner/test-open-cv/main.py", line 2, in <module>
    import cv2 as cv
  File "/home/runner/test-open-cv/.pythonlibs/lib/python3.10/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/home/runner/test-open-cv/.pythonlibs/lib/python3.10/site-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
  File "/nix/store/xf54733x4chbawkh1qvy9i1i4mlscy1c-python3-3.10.11/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

No system dependencies or replit.nix file were installed or generated. (Automatic system dependencies did work for scipy on the same repl.)

1 Like

https://blog.replit.com/system-dependencies-python

Thanks for all the feedback so far!

3 Likes