Installing titiler with poetry?

[tmcw]

I'm a newcomer to "modern Python package management" and as far as I can tell, to get reproducible builds, the right move is to use poetry. That has been a confusing experience with titiler:

With this tiny pyproject.toml configuration for poetry:

[tool.poetry]
name = "placemark-tiler"
version = "0.1.0"
description = ""
authors = ["Tom MacWright <tom@macwright.com>"]
readme = "README.md"
packages = [{include = "placemark_tiler"}]

[tool.poetry.dependencies]
python = "^3.10"
uvicorn = "^0.18.3"
titiler-application = "^0.7"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I get this build failure:

Oct 12 04:18:40 PM  ==> Running build command './build.sh'...
Oct 12 04:18:42 PM  Installing dependencies from lock file
Oct 12 04:18:43 PM  
Oct 12 04:18:43 PM    SolverProblemError
Oct 12 04:18:43 PM  
Oct 12 04:18:43 PM    Because no versions of titiler-application match >=0.7,<0.7.1 || >0.7.1,<0.8
Oct 12 04:18:43 PM     and titiler-application (0.7.1) depends on titiler.mosaic (>=0.7,<0.8), titiler-application (>=0.7,<0.8) requires titiler.mosaic (>=0.7,<0.8).
Oct 12 04:18:43 PM    So, because no versions of titiler.mosaic match >=0.7,<0.8
Oct 12 04:18:43 PM     and placemark-tiler depends on titiler-application (^0.7), version solving failed.
Oct 12 04:18:43 PM  
Oct 12 04:18:43 PM    at /home/render/.python-poetry/venv/lib/python3.7/site-packages/poetry/puzzle/solver.py:241 in _solve
Oct 12 04:18:43 PM        237│             packages = result.packages
Oct 12 04:18:43 PM        238│         except OverrideNeeded as e:
Oct 12 04:18:43 PM        239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
Oct 12 04:18:43 PM        240│         except SolveFailure as e:
Oct 12 04:18:43 PM      → 241│             raise SolverProblemError(e)
Oct 12 04:18:43 PM        242│
Oct 12 04:18:43 PM        243│         results = dict(
Oct 12 04:18:43 PM        244│             depth_first_search(
Oct 12 04:18:43 PM        245│                 PackageNode(self._package, packages), aggregate_package_nodes

This raises a lot of weird, weird questions. For example:

because no versions of titiler.mosaic match >=0.7,<0.8

Don't they? There's definitely titiler.mosaic 0.7.1 on PyPi, right?

Because no versions of titiler-application match >=0.7,<0.7.1 || >0.7.1,<0.8

This version query is really odd, it's searching for something like "every version that's higher than 0.7.1 and less than 8", which… why? I'm not seeing any semver that would require that.

---

So… any ideas, Pythonistas? Is poetry the right tool to use if I want something like the npm or yarn "lockfile-based reproducible builds" experience? Is there something I'm missing that is making this oddly difficult to get started with?

[vincentsarago]

Hi @tmcw

Thanks for starting this discussion.

I'm sorry for this. The state of titiler self dependency is really frustration. So in Short:

• titiler.application depends on titiler.core and titiler mosaic (>=0.7,<0.8)
• titiler.mosaic depends on titiler.core>=0.7,<0.8

This is a bit weird, we tried to go with monorepo and multiple package with titiler (thanks @kylebarron) but I'm not sure it was the best move (or at least the execution is not perfect).

I'm not sure why Poetry is complaining (I'm not a poetry user because of this kinda of issue) but I'll ask around.

[tmcw]

Erg, yeah - this is working locally, failing in production. Looking into it.

[kylebarron]

failing in production

Does it fail to install from the lockfile you save locally? Are you using a different machine type in prod? Might just be struggling to build from source if wheels don't exist for some projects

[tmcw]

And the solution seems to be… poetry < 1.2.0 can't install lockfiles created by poetry >= 1.2.0 which involve packages with dot characters! Prod was using an old version (1.1.14) and I was using 1.2.2 locally.

The lockfile format changed between 1.1.0 and 1.2.0, but poetry doesn't include its own version in the file (bummer!) and didn't preserve forwards compatibility for the file, so, anyway… there's an upstream issue that has more details: python-poetry/poetry#6593

Poetry 1.2 changed the way it normalizes . characters in way that's incompatible with 1.1's installation method and consistently produces this error. Learn something new every day!

[kylebarron]

Sorry Tom 😭 I've hit that same issue before where Poetry 1.2 changed some lockfile behavior but didn't think to mention it. Locking the poetry version is definitely important for sanity

[kylebarron]

From python-poetry/poetry#6593 (comment)

In most cases, installing a lock file generated by Poetry 1.2 with Poetry 1.1 should work just fine

Interesting... Maybe they think the final 1.2 version solved some of the issue

[vincentsarago]

FYI: for your tiler you may want to start with only titiler.core. titiler.application is a demo that use core and mosaic to build a full application

see: https://developmentseed.org/titiler/intro/#customized-minimal-app

[vincentsarago]

second note: titiler doesn't official support python 3.10 (it will in couple weeks)

It should work fine but I haven't run CI yet

[tmcw]

Okay, yeah, that seems like the way to do it, thx! That diamond-dependency problem is apparently too much for poetry's solver.

[vincentsarago]

it worked for me 🙈, but using python 3.9 and

virtualenv -p /opt/homebrew/bin/python3.9 venv
source venv/bin/activate     
pip install poetry 

poetry install 

Package operations: 46 installs, 1 update, 0 removals

  • Installing click (8.1.3)
  • Installing numpy (1.23.4)
  • Installing affine (2.3.1)
  • Installing click-plugins (1.1.1)
  • Installing cligj (0.7.2)
  • Installing jmespath (1.0.1)
  • Installing python-dateutil (2.8.2)
  • Updating setuptools (65.3.0 -> 65.4.1)
  • Installing snuggs (1.4.7)
  • Installing botocore (1.27.89)
  • Installing h11 (0.14.0)
  • Installing rasterio (1.3.2)
  • Installing sniffio (1.3.0)
  • Installing typing-extensions (4.4.0)
  • Installing anyio (3.6.1)
  • Installing httpcore (0.13.2)
  • Installing pydantic (1.10.2)
  • Installing pyproj (3.4.0)
  • Installing rfc3986 (1.5.0)
  • Installing rio-mucho (1.0.0)
  • Installing s3transfer (0.6.0)
  • Installing boto3 (1.24.89)
  • Installing cachetools (5.2.0)
  • Installing httpx (0.18.1)
  • Installing markupsafe (2.1.1)
  • Installing mercantile (1.2.1)
  • Installing morecantile (3.1.2)
  • Installing numexpr (2.8.3)
  • Installing pystac (1.6.1)
  • Installing rio-color (1.0.4)
  • Installing starlette (0.20.4)
  • Installing fastapi (0.85.0)
  • Installing geojson-pydantic (0.4.3)
  • Installing jinja2 (3.1.2)
  • Installing pygeos (0.13)
  • Installing rio-tiler (3.1.6)
  • Installing simplejson (3.17.6)
  • Installing supermercado (0.2.0)
  • Installing cogeo-mosaic (4.1.0)
  • Installing cramjam (2.5.0)
  • Installing titiler-core (0.7.1)
  • Installing python-dotenv (0.21.0)
  • Installing rio-cogeo (3.4.1)
  • Installing starlette-cramjam (0.3.1)
  • Installing titiler-mosaic (0.7.1)
  • Installing titiler-application (0.7.1)
  • Installing uvicorn (0.18.3)

Installing the current project: placemark-tiler (0.1.0)