tiledbsoma 1.8.0 pre-check #91

johnkerl · 2024-02-28T16:10:33Z

See also #93 which addresses issues which have happened since 1.7.2 -- things involving our dependencies -- which have nothing to do with 1.8.0

See also single-cell-data/TileDB-SOMA#2219 for tracking.

johnkerl · 2024-02-28T16:13:10Z

[sc-41030]

shortcut-integration · 2024-02-28T16:13:15Z

This pull request has been linked to Shortcut Story #41030: Update projects to TileDB 2.20.

jdblischak · 2024-02-28T17:17:52Z

Currently failing with similar conda solver errors as we previously identified in #89 (comment)

We are getting conda solver errors in multiple feedstocks, so there must be some systemic problem. I'm hoping that this commit merged earlier this morning (which migrates to libgoogle-cloud 2.21) will help ease the solver problems 🤞

johnkerl · 2024-02-28T19:41:16Z

I'm hoping that this commit merged earlier this morning (which migrates to libgoogle-cloud 2.21) will help ease the solver problems 🤞

@jdblischak are we waiting for some sort of propagation? 🤔

jdblischak · 2024-02-28T19:52:12Z

are we waiting for some sort of propagation?

Yes, I'm hoping that maybe the new tiledb 2.20.1 binaries that were built against libgoogle-cloud 2.21 might help with the current solver issue. They were uploaded more than 2 hours ago, so should be available now

https://anaconda.org/conda-forge/tiledb/files?version=2.20.1

(note that the build numbers are weird (two different sets with build number 1) because of all the merge commits we've done recently in tiledb-feedstock. Fortunately the migration to libgoogle-cloud 2.21 also changed the build hash, so new binaries were uploaded)

jdblischak · 2024-02-28T19:57:04Z

Welp that didn't fix it. The latest failed CI run used the latest tiledb built against libgoogle-cloud 2.21

    libgoogle-cloud:          2.21.0-h31df0ca_1           conda-forge

…nda-forge-pinning 2024.02.28.18.36.29

jdblischak · 2024-02-29T15:08:56Z

The solver error from the latest run was decently informative. We know it has to do with libabseil, libcurl, and libgoogle-cloud-storage

    Could not solve for environment specs
    The following packages are incompatible
    ├─ libabseil ==20240116.1 cxx17_h59595ed_1 is requested and can be installed;
    ├─ libcurl ==8.5.0 hca28451_0 is requested and can be installed;
    ├─ libgoogle-cloud-storage ==2.21.0 hc7a4891_2 is requested and can be installed;
    └─ libtiledbsoma ==1.8.0 hd763302_0 is not installable because it requires
       └─ tiledb >=2.20.1,<2.21.0a0  but there are no viable options
          ├─ tiledb 2.20.1 would require
          │  └─ libcurl >=7.88.1,<8.0a0 , which conflicts with any installable versions previously reported;
          ├─ tiledb 2.20.1 would require
          │  └─ libgoogle-cloud-storage >=2.17.0,<2.18.0a0 , which conflicts with any installable versions previously reported;
          └─ tiledb 2.20.1 would require
             └─ libabseil >=20230802.1,<20230803.0a0 , which conflicts with any installable versions previously reported.

johnkerl · 2024-02-29T15:41:21Z

https://anaconda.org/conda-forge/google-cloud-cpp/files updated 4 days ago ... 🤔

johnkerl · 2024-02-29T15:58:02Z

Weird thing is 2.17 doesn't exist at all at https://anaconda.org/conda-forge/google-cloud-cpp/files ... 🤔

jdblischak · 2024-02-29T16:06:11Z

https://anaconda.org/conda-forge/libgoogle-cloud-storage/files?version=2.17.0

johnkerl · 2024-02-29T16:09:55Z

OK I may be mixing google-cloud-cpp and google-cloud-storage ... OTOH from the last raw log:

johnkerl@exade[prod][][~]$ grep -C1 google-cloud Desktop/raw.txt
2024-02-28T22:31:52.2156877Z     libgomp:                  13.2.0-h807b86a_5           conda-forge
2024-02-28T22:31:52.2157237Z     libgoogle-cloud:          2.21.0-h31df0ca_1           conda-forge
2024-02-28T22:31:52.2157609Z     libgoogle-cloud-storage:  2.21.0-hc7a4891_1           conda-forge
2024-02-28T22:31:52.2157976Z     libgrpc:                  1.60.1-h74775cd_0           conda-forge
--
2024-02-28T22:34:23.3632616Z     libgomp:                   13.2.0-h807b86a_5           conda-forge
2024-02-28T22:34:23.3633104Z     libgoogle-cloud:           2.17.0-h31df0ca_2           conda-forge
2024-02-28T22:34:23.3634545Z     libgoogle-cloud-storage:   2.17.0-hc7a4891_2           conda-forge
2024-02-28T22:34:23.3635111Z     libgrpc:                   1.60.1-h74775cd_0           conda-forge
--
2024-02-28T22:36:32.5382008Z     libgomp:                   13.2.0-h807b86a_5           conda-forge
2024-02-28T22:36:32.5382437Z     libgoogle-cloud:           2.17.0-h31df0ca_2           conda-forge
2024-02-28T22:36:32.5382872Z     libgoogle-cloud-storage:   2.17.0-hc7a4891_2           conda-forge
2024-02-28T22:36:32.5383298Z     libgrpc:                   1.60.1-h74775cd_0           conda-forge
--
2024-02-28T22:38:44.8621716Z     ├─ libcurl ==8.5.0 hca28451_0 is requested and can be installed;
2024-02-28T22:38:44.8622037Z     ├─ libgoogle-cloud-storage ==2.21.0 hc7a4891_2 is requested and can be installed;
2024-02-28T22:38:44.8622353Z     └─ libtiledbsoma ==1.8.0 hd763302_0 is not installable because it requires
--
2024-02-28T22:38:44.8623815Z           ├─ tiledb 2.20.1 would require
2024-02-28T22:38:44.8624167Z           │  └─ libgoogle-cloud-storage >=2.17.0,<2.18.0a0 , which conflicts with any installable versions previously reported;
2024-02-28T22:38:44.8636944Z           └─ tiledb 2.20.1 would require

🤔

johnkerl · 2024-02-29T22:15:08Z

[sc-42200]

shortcut-integration · 2024-02-29T22:15:11Z

This pull request has been linked to Shortcut Story #42200: tiledbsoma 1.8.0.

ihnorton · 2024-02-29T22:23:15Z

recipe/meta.yaml

@@ -90,6 +91,7 @@ outputs:
        - pyarrow <13.0 # [osx]
        - pyarrow       # [not osx]
        - pyarrow-hotfix
+        - libgoogle-cloud 2.12.*


Isn't this supposed to be 2.21?

(actually, why are we pinning this at all?)

It is not

See Slack

I'll add some details here momentarily

2.12 is absolutely and unequivocally needed for core 2.19 / tiledbsoma 1.7:
#93 (comment)

I'm confirming now for this PR 👀

(actually, why are we pinning this at all?)

It's a hacky workaround we are applying to unblock tiledbsoma. Originally discovered in #89, this solver error came out of nowhere. I suspect it has to do with the recent conda-forge switch from google-cloud-cpp to libgoogle-cloud, but I've been too busy with the libcurl 7 stuff to figure out exactly what is going on.

Ok, carry on - thanks.

For this PR 2.12 doesn't work although it did for #93:
https://gist.github.com/johnkerl/7812fb057db778d8bfb17b424dec6ead

Trying 2.17 locally ... 🤞

Agreed. Based on the error, I think libgoogle-cloud 2.17 should work

Could not solve for environment specs The following packages are incompatible ├─ libgoogle-cloud ==2.17.0 h31df0ca_2 is requested and can be installed; └─ libtiledbsoma ==1.8.0 hd763302_0 is not installable because it requires └─ libgoogle-cloud >=2.12.0,<2.13.0a0 , which conflicts with any installable versions previously reported.

libgoogle-cloud 2.17 fixed the conda solver error. However, only the linux-64 build is green

jdblischak · 2024-03-01T02:40:00Z

The osx-64 build is now failing because the r-tiledbsoma compilation is failing with a bunch of macOS SDK errors. Maybe we need to adjust MACOSX_SDK_VERSION and/or MACOSX_DEPLOYMENT_TARGET?

tiledbsoma-feedstock/recipe/conda_build_config.yaml

Lines 1 to 4 in 6663e51

    
           MACOSX_SDK_VERSION:  # [osx and x86_64] 
        
             - 10.14            # [osx and x86_64] 
        
           MACOSX_DEPLOYMENT_TARGET:  # [osx and x86_64] 
        
             - "10.14"                # [osx and x86_64]

Here is the final part of the errors:

/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_build_env/bin/../include/c++/v1/__filesystem/path.h:780:25: note: 'operator basic_string' has been explicitly marked unavailable here
  _LIBCPP_HIDE_FROM_ABI operator string_type() const { return __pn_; }
                        ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/tiledbsoma_types.h:21:
In file included from /Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/tiledbsoma:51:
/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/soma/soma_collection.h:105:44: error: 'path' is unavailable: introduced in macOS 10.15 - see https://conda-forge.org/docs/maintainer/knowledge_base.html#newer-c-features-with-old-sdk
              std::string(std::filesystem::path(uri).filename()),  // group name
                                           ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_build_env/bin/../include/c++/v1/__filesystem/path.h:442:24: note: 'path' has been explicitly marked unavailable here
class _LIBCPP_TYPE_VIS path {
                       ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/etc/Makeconf:200: RcppExports.o] Error 1
ERROR: compilation failed for package ‘tiledbsoma’
* removing ‘/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/library/tiledbsoma’

In the osx-arm64 build, r-tiledbsoma compiles fine. The failure is due to tiledbsoma-py being unable to find libtiledbsoma.so in the conda env, and then failing to build it from source:

running build_ext
Checking: /Users/runner/miniforge3/conda-bld/tiledbsoma_1709247012541/work/dist/lib/libtiledbsoma.dylib exists: False
dlopen(libtiledbsoma.dylib, 6): image not found
Build with TileDB: /Users/runner/miniforge3/conda-bld/tiledbsoma_1709247012541/work/dist
Building Release build
./bld: line 88: cmake: command not found

My guess is that this might be due to single-cell-data/TileDB-SOMA#1937 since it updated how tiledbsoma-py interacts with an existing libtiledbsoma.so

jdblischak · 2024-03-01T02:46:08Z

From the nightly feedstock builds, the R compilation error on osx-64 started on Feb 12th. And the osx-arm64 builds started failing on Feb 16th, so I am more confident that it was caused by single-cell-data/TileDB-SOMA#1937, which was merged earlier that day in single-cell-data/TileDB-SOMA@f4999d8

johnkerl · 2024-03-01T22:47:41Z

@jdblischak FYI - libgoogle-cloud 2.17.* no longer solves -- this happened today -- I've removed that line

I've confirmed this locally (for linux) as well

johnkerl · 2024-03-01T23:12:15Z

@jdblischak @ihnorton linux is once again green.

OSX is not; and I've switched from 10.14 to 10.15 in an attempt to rid errors of the form

2024-03-01T22:51:51.7619640Z /Users/runner/miniforge3/conda-bld/tiledbsoma_1709332344079/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/soma/soma_sparse_ndarray.h:121:54: error: '~path' is unavailable: introduced in macOS 10.15 - see https://conda-forge.org/docs/maintainer/knowledge_base.html#newer-c-features-with-old-sdk
2024-03-01T22:51:51.7620440Z               std::string(std::filesystem::path(uri).filename()),  // array name
2024-03-01T22:51:51.7620860Z                                                      ^
2024-03-01T22:51:51.7621620Z /Users/runner/miniforge3/conda-bld/tiledbsoma_1709332344079/_build_env/bin/../include/c++/v1/__filesystem/path.h:505:3: note: '~path' has been explicitly marked unavailable here
2024-03-01T22:51:51.7622230Z   ~path() = default;
2024-03-01T22:51:51.7622590Z   ^

which is weird b/c of the error

error: '~path' is unavailable: introduced in macOS 10.15

and I am using 10.15 :(

johnkerl · 2024-03-01T23:28:04Z

... raw logs reveal 10.14 still appearing in log output. Trying a rerender ...

…nda-forge-pinning 2024.03.01.21.24.14

johnkerl · 2024-03-01T23:29:52Z

d416ebc 🤞

https://conda-forge.org/docs/maintainer/knowledge_base/#newer-c-features-with-old-sdk

jdblischak · 2024-03-04T16:34:55Z

Ugh, the osx-64 one failed with a spurious download error. That is the main one we need feedback on. I'm going to cancel and restart the CI jobs

johnkerl · 2024-03-05T20:28:18Z

See also single-cell-data/TileDB-SOMA#2210

johnkerl · 2024-03-06T15:01:13Z

CI is green; this PR has done its job per our established procedure

See also notes on single-cell-data/TileDB-SOMA#2210 that the cmake dependency in the build line here is a workaround not a fix -- build-from-source should not be happening in the first place

johnkerl · 2024-03-06T19:16:11Z

More advice from @jdblischak

jdblischak · 2024-03-06T20:11:50Z

The --no-build-isolation flag is working as intended. pip is no longer installing the build deps

xref: #95 (comment)

/home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/python -m pip install --no-deps --no-build-isolation . -vv
Using pip 24.0 from /home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.10/site-packages/pip (python 3.10)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-_creqxao
Initialized build tracking at /tmp/pip-build-tracker-_creqxao
Created build tracker: /tmp/pip-build-tracker-_creqxao
Entered build tracker: /tmp/pip-build-tracker-_creqxao
Created temporary directory: /tmp/pip-install-qfsfg8yk
Created temporary directory: /tmp/pip-ephem-wheel-cache-ly63dwyh
Processing /home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python
  Added file:///home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python to build tracker '/tmp/pip-build-tracker-_creqxao'
  Created temporary directory: /tmp/pip-modern-metadata-3e9ncd7i
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info
  writing /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'py.typed'
  no previously-included directories found matching 'dist_links/libtiledbsoma/test/__pycache__'
  writing manifest file '/tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma-1.5.0rc0.post226.dev267876719.dist-info'
  Preparing metadata (pyproject.toml): finished with status 'done'
  Source in /home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python has version 1.5.0rc0.post226.dev267876719, which satisfies requirement tiledbsoma==1.5.0rc0.post226.dev267876719 from file:///home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python
  Removed tiledbsoma==1.5.0rc0.post226.dev267876719 from file:///home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python from build tracker '/tmp/pip-build-tracker-_creqxao'
Created temporary directory: /tmp/pip-unpack-q9ol_44z
Building wheels for collected packages: tiledbsoma
  Created temporary directory: /tmp/pip-wheel-shq_ssco
  Destination directory: /tmp/pip-wheel-shq_ssco
  Building wheel for tiledbsoma (pyproject.toml): started
  Running command Building wheel for tiledbsoma (pyproject.toml)
  running bdist_wheel
  Checking: /home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libtiledbsoma.so exists: True
  running build
  running build_py

tiledbsoma 1.0.8 pre-check

0fd006c

johnkerl changed the title ~~tiledbsoma 1.0.8 pre-check~~ tiledbsoma 1.8.0 pre-check Feb 28, 2024

jdblischak mentioned this pull request Feb 28, 2024

Troubleshoot unexpected conda solver error #92

Closed

johnkerl mentioned this pull request Feb 28, 2024

Rerender, and fix libgoogle-cloud issues #93

Merged

MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.31.1, and co…

a7f6003

…nda-forge-pinning 2024.02.28.18.36.29

johnkerl marked this pull request as ready for review February 29, 2024 21:52

johnkerl requested review from gspowley, jdblischak, mlin and Shelnutt2 as code owners February 29, 2024 21:52

Apply findings from #93

2812f82

johnkerl force-pushed the kerl/tiledbsoma-1.8.0-pre-check branch from a2dd497 to 2812f82 Compare February 29, 2024 22:16

ihnorton approved these changes Feb 29, 2024

View reviewed changes

ihnorton reviewed Feb 29, 2024

View reviewed changes

try libgoogle-cloud 2.17

07c21bd

johnkerl added 3 commits March 1, 2024 16:47

try macosx 10.14 -> 10.15 like some other feedstocks do

6f2562d

iterating

d299856

typofix

f4f7ff4

MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.31.1, and co…

d416ebc

…nda-forge-pinning 2024.03.01.21.24.14

johnkerl marked this pull request as ready for review March 2, 2024 03:13

Use conda-forge workaround for newer C++ features on macOS

e2ad580

https://conda-forge.org/docs/maintainer/knowledge_base/#newer-c-features-with-old-sdk

jdblischak and others added 3 commits March 4, 2024 11:58

The macOS SDK errors are from the r-tiledbsoma build with r-base 4.3

cc23420

try @jdblischak items from #95

7bab53b

update to today's tip of release-1.8 banch

fb14260

This was referenced Mar 5, 2024

[ci] Run python-so-copying.yml on Linux and Mac both [WIP] single-cell-data/TileDB-SOMA#2208

Closed

[release-1.8] Revert #1937 single-cell-data/TileDB-SOMA#2209

Closed

[ci] dylib-copy logic appears to be broken on MacOS single-cell-data/TileDB-SOMA#2210

Closed

add cmake to build dep

3223029

johnkerl closed this Mar 6, 2024

johnkerl mentioned this pull request Mar 6, 2024

Link-tracking for TileDB-SOMA 1.7.3 and 1.8.0 single-cell-data/TileDB-SOMA#2219

Closed

more advice from @jdblischak

ac62e59

johnkerl reopened this Mar 6, 2024

update git hash to tip of release-1.8

74ca0a3

jdblischak mentioned this pull request Mar 6, 2024

[python] Fix and test setup.py flags to external shared objects single-cell-data/TileDB-SOMA#2221

Merged

more branch-tip

33b438b

johnkerl closed this Mar 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tiledbsoma 1.8.0 pre-check #91

tiledbsoma 1.8.0 pre-check #91

johnkerl commented Feb 28, 2024 •

edited

Loading

johnkerl commented Feb 28, 2024

shortcut-integration bot commented Feb 28, 2024

jdblischak commented Feb 28, 2024

johnkerl commented Feb 28, 2024 •

edited

Loading

jdblischak commented Feb 28, 2024

jdblischak commented Feb 28, 2024

jdblischak commented Feb 29, 2024

johnkerl commented Feb 29, 2024

johnkerl commented Feb 29, 2024

jdblischak commented Feb 29, 2024

johnkerl commented Feb 29, 2024

johnkerl commented Feb 29, 2024

shortcut-integration bot commented Feb 29, 2024

ihnorton Feb 29, 2024

ihnorton Feb 29, 2024

johnkerl Feb 29, 2024

johnkerl Feb 29, 2024

johnkerl Feb 29, 2024

jdblischak Feb 29, 2024

ihnorton Feb 29, 2024

johnkerl Feb 29, 2024

jdblischak Feb 29, 2024

jdblischak Mar 1, 2024

jdblischak commented Mar 1, 2024

jdblischak commented Mar 1, 2024

johnkerl commented Mar 1, 2024

johnkerl commented Mar 1, 2024 •

edited

Loading

johnkerl commented Mar 1, 2024

johnkerl commented Mar 1, 2024

jdblischak commented Mar 4, 2024

johnkerl commented Mar 5, 2024

johnkerl commented Mar 6, 2024

johnkerl commented Mar 6, 2024

jdblischak commented Mar 6, 2024

tiledbsoma 1.8.0 pre-check #91

tiledbsoma 1.8.0 pre-check #91

Conversation

johnkerl commented Feb 28, 2024 • edited Loading

johnkerl commented Feb 28, 2024

shortcut-integration bot commented Feb 28, 2024

jdblischak commented Feb 28, 2024

johnkerl commented Feb 28, 2024 • edited Loading

jdblischak commented Feb 28, 2024

jdblischak commented Feb 28, 2024

jdblischak commented Feb 29, 2024

johnkerl commented Feb 29, 2024

johnkerl commented Feb 29, 2024

jdblischak commented Feb 29, 2024

johnkerl commented Feb 29, 2024

johnkerl commented Feb 29, 2024

shortcut-integration bot commented Feb 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdblischak commented Mar 1, 2024

jdblischak commented Mar 1, 2024

johnkerl commented Mar 1, 2024

johnkerl commented Mar 1, 2024 • edited Loading

johnkerl commented Mar 1, 2024

johnkerl commented Mar 1, 2024

jdblischak commented Mar 4, 2024

johnkerl commented Mar 5, 2024

johnkerl commented Mar 6, 2024

johnkerl commented Mar 6, 2024

jdblischak commented Mar 6, 2024

johnkerl commented Feb 28, 2024 •

edited

Loading

johnkerl commented Feb 28, 2024 •

edited

Loading

johnkerl commented Mar 1, 2024 •

edited

Loading