[BLOG] Add libsf_error_state: SciPy's first shared library #892

steppi · 2024-12-17T04:05:20Z

Sorry for accidentally targeting main with my first PR. Copied from before:

This is nearly done so I figured it was time to make a PR here. Irwin Zaid, who was involved in the work described has read this draft and given a thumbs up (after making a couple small suggestions).

I still need to add footnotes and write the conclusion section at the end, which currently has section header ##Reflections. It looks like I'll also need to go through and confirm the text styling bullet points.

Text styling

The blog is written with plain language (where relevant).
If there are headers, they use the proper header tags in order to do so (with only one level-one header).
All links describe where they link to (for example, check the Quansight labs website).
Any kind of styling that the author uses (for example, bold for emphasis) is consistent throughout the blog.

Non-text contents

Blog post featured image is in PNG or JPEG format, not SVG.
All content is represented as text (for example, images need alt text and videos need captions or descriptive transcripts).
If there are emojis, there are not more than three in a row.
Don't use flashing gifs or videos.
If it were to be read as plain text, the blog still makes sense and no information is missing.

vercel · 2024-12-17T04:05:24Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
labs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 20, 2024 4:45pm

rgommers

Great write-up Albert! I like the structure of the story.

rgommers · 2024-12-19T10:45:42Z

apps/labs/posts/libsf-error-state.md

+code, since one need not worry about the thread-safety of wrapped compiled libraries.
+
+By wrapping, filling in gaps, and providing user friendly Python APIs to a wide range of battle tested, permissively
+licensed and public domain scientific software tools—SciPy's founding developers [] were able to kickstart the growth of


some spurious brackets here

Thanks Ralf, those were the places where I meant to go back and add footnotes. They're added now

rgommers · 2024-12-19T10:46:47Z

apps/labs/posts/libsf-error-state.md

+complexity SciPy tries to hide from users, while previewing some exciting developments in
+[`scipy.special`](https://docs.scipy.org/doc/scipy/reference/special.html).
+
+Python has become wildly popular as a scripting language for scientific and other data intensive applications. It owes


nit: I'd remove "scripting" - it doesn't apply well to how many devs use Python, and is often read as a negative adjective.

rgommers · 2024-12-19T10:47:50Z

apps/labs/posts/libsf-error-state.md

+
+By wrapping, filling in gaps, and providing user friendly Python APIs to a wide range of battle tested, permissively
+licensed and public domain scientific software tools—SciPy's founding developers [] were able to kickstart the growth of
+the Scientific Python ecosystem. There are now worlds of scientific and data intensive software available in


should be "data-intensive" with a dash I think?

rgommers · 2024-12-19T10:53:00Z

apps/labs/posts/libsf-error-state.md

+
+## NumPy Universal functions
+
+A NumPy [`ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) represents an arbitrary


This transition is a little abrupt. I'd suggest guiding the reader a bit by putting a sentence above the section header like "Our simple feature here is correct error handling for SciPy's special functions. We'll start with how special functions are implemented with NumPy ufuncs, and from there to how error handling is designed and finally implemented with the help of a shared library." (quick sketch, the actual flow may be a bit different, I haven't read to the end yet).

rgommers · 2024-12-19T10:55:29Z

apps/labs/posts/libsf-error-state.md

+
+To create a ufunc for a mathematical function, one needs a scalar implementation of this function, known as a scalar
+kernel, that's written in a compiled language. Until recently, `scipy.special` had scalar kernels written in all of C,
+C++, Fortran 77, and Cython. In August of 2023, Irwin Zaid (izaid) at Oxford proposed rewriting all of the scalar


change "izaid" to @izaid with a link to his Github profile?

rgommers · 2024-12-19T11:27:42Z

apps/labs/posts/libsf-error-state.md

+free-threaded Python across the ecosystem for much of this past year.
+
+In a curious reversal, it is now (the `win32` flavor of) MinGW that is causing trouble due to lack of proper threading
+support [].


empty brackets. I think the sentence should be continued with something like "since MinGW does not support thread local variables".

I was basing it on this from Edgar

It seems that using std::mutex also causes the same MinGW threading issues to surface.

and part of your reply

MinGW-w64 comes in two build flavors: posix and win32. The posix builds have threading support, the win32 ones don't*

It seemed like something deeper might be going on, since even the mutex isn't working. I don't really understand the underlying issue though.

I don't think there's a need to explain in detail, just the basic reason (MinGW doesn't have support).

I can't claim to fully understand it either, MinGW stuff is usually quite arcane and requires digging through lots of old email threads.

rgommers · 2024-12-19T11:28:09Z

apps/labs/posts/libsf-error-state.md

+Now that the dust has mostly settled, it's valuable to look back and try to judge whether we made the right choice. Over
+the entire timeframe, the primary goal for Irwin and I was always to make as much of SciPy special available on GPUs as
+possible—with secondary goals of simplifying SciPy special's build process and improving the scalar kernel codebase. The
+story of `libsf_error_state` is that of a side quest—a story of needing to fix something we broke in pursuit of bigger


spaces around brackets here

rgommers · 2024-12-19T11:28:24Z

apps/labs/posts/libsf-error-state.md

+story of `libsf_error_state` is that of a side quest—a story of needing to fix something we broke in pursuit of bigger
+things—and the challenges we faced and brought to others because we underestimated the difficulty of our chosen
+solution. Having a single array for managing shared state between the extension modules appeals to a desire to have a
+single source of truth [], but looking back, it would have taken much less time to have each extension module carry its


empty brackets

rgommers · 2024-12-19T11:29:55Z

apps/labs/posts/libsf-error-state.md

+[`scipy.special.cython_special`](https://docs.scipy.org/doc/scipy/reference/special.cython_special.html), a module for
+calling scalar versions of special function from Cython. To quote:
+
+"To make the necessary modifications to cython_special error handling we need to instead share a single global variable


typeset cython_special as code here?

rgommers · 2024-12-19T11:30:11Z

apps/labs/posts/libsf-error-state.md

+calling scalar versions of special function from Cython. To quote:
+
+"To make the necessary modifications to cython_special error handling we need to instead share a single global variable
+between the two extension modules. On gcc we could do this with -rpath, but coming up with a portable solution seems


GCC in capitals and -rpath as code?

This is a verbatim quote, so I didn't want to edit it.

steppi · 2024-12-19T17:02:33Z

Thanks @rgommers for all of the suggestions. Me not understanding why the inplace build works but not dev.py seems to be most in need of fixing. I'll look into understanding what's really going on. The empty brackets were places to remind me where to put the footnotes later. I should have mentioned that in my PR message. I've added footnotes now, and a conclusion section, and will go through and implement all of your suggestions.

steppi · 2024-12-19T19:23:57Z

@rgommers, I think I have almost everything addressed now. A couple questions though:

I have a bunch of footnotes that are just links. I did this to better follow this advice above "All links describe where they link to (for example, check the Quansight labs website)." This is for cases where I just want to point to a place to get more information. In each case I did this, the url clearly describes where it links to. Is this the right thing to do? I could also just remove the links entirely and trust readers to look things up if needed, but my thought was that explicitly linking is also vouching for the linked content.

There's another issue where I was putting links in places where something should be formatted as code. This seems to not actually work, they end up not really looking like links to me. I think I'll just take those out.

An example:

[`ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html)

this looks fine in github

ndarray

but in the labs blog:

and the color doesn't change when hovering.

This also doesn't describe what is linked to, but I thought maybe the convention of having all such links point to the official documentation would be clear enough.

steppi · 2024-12-19T19:37:10Z

I have a bunch of footnotes that are just links. I did this to better follow this advice above "All links describe where they link to (for example, check the Quansight labs website)."

Looking at some other posts, it seems pretty common just to have links directly in the text associated to a term or phrase without explicitly saying what is linked to, and also to just have link text that is formatted with backticks, so I guess it's fine to go back to that. Having links as footnotes seems like a worse user experience.

steppi · 2024-12-19T21:02:10Z

I think all of the feedback so far has been addressed now.

rgommers · 2024-12-19T21:08:22Z

Yes, I think direct links are fine - footnotes are a bit more awkward imho.

rgommers · 2024-12-19T21:10:49Z

Thanks Albert. I'll try to have another look tomorrow.

steppi · 2024-12-19T23:45:23Z

Thanks Ralf!

rgommers

Almost there! A few more comments before the weekend.

rgommers · 2024-12-20T14:18:47Z

apps/labs/posts/libsf-error-state.md

+## NumPy Universal functions
+
+The simple feature in question is error handling for mathematical functions in `scipy.special` that operate
+elementwise over NumPy arrays. Let's look into this more closely.


minor: looks like this should be above the section header

rgommers · 2024-12-20T14:19:39Z

apps/labs/posts/libsf-error-state.md

+
+Here's the ufunc [`np.sqrt`](https://numpy.org/doc/stable/reference/generated/numpy.sqrt.html) in action:
+
+```


add python here for syntax highlighting (also in the next code blocks)

rgommers · 2024-12-20T14:39:56Z

apps/labs/posts/libsf-error-state.md

+```
+
+For large arrays, the speedup from applying `np.sqrt` over an `ndarray` rather than
+[`math.sqrt`](https://docs.python.org/3/library/math.html#math.sqrt) over each element of a list is significant. On my


minor: I would not use links for common things like math.sqrt, float, ValueError - that's more distracting than helpful in my opinion. Less well known things like np.errstate do seem fine as links.

rgommers · 2024-12-20T14:42:02Z

apps/labs/posts/libsf-error-state.md

+We saw three options:
+
+1. Have the Python interface for modifying the error handling state (`special.errstate` and
+   [`special.seterr`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.seterr.html)) update the state


special.seterr was already linked higher up, so please remove the link here

rgommers · 2024-12-20T14:56:50Z

apps/labs/posts/libsf-error-state.md

+
+Even after this, I was still seeing the same error. I asked Ralf about it, and found what happens is that `pip`
+goes through [meson-python](https://github.com/mesonbuild/meson-python) while `dev.py` uses Meson directly.
+`meson-python` takes care to set the [`RPATH`](https://en.wikipedia.org/wiki/Rpath) for each extension module.


I'll try to rephrase this before merging, since it's a little more subtle and I'd prefer for it to be as correct as possible.

rgommers · 2024-12-20T16:07:46Z

apps/labs/posts/libsf-error-state.md

+a Clang compiler driver that aims for compatability with MSVC. Clang-cl is used for Windows builds in conda-forge and we had run into
+another platform specific difference.
+
+Fortunately, h-vetinari knew what the problem was. On Linux, Mac OS (and Windows while using MinGW), symbols from shared


nit: "Mac OS" -> "macOS"

rgommers · 2024-12-20T16:08:21Z

apps/labs/posts/libsf-error-state.md

+
+He had a recipe ready to use: defining and using macros which conditionally compiled to the right thing depending on their context.
+
+```C++


highlighting doesn't work, so this probably should be cpp

rgommers · 2024-12-20T16:10:02Z

apps/labs/posts/libsf-error-state.md

+[free-threaded Python team](https://labs.quansight.org/blog/free-threaded-python-rollout) — to ensure
+thread safety by declaring the array `sf_error_actions` thread local. This eliminates the data-race by making it so each
+thread gets its own separate copy of the array. Edgar and the others on the free-threaded Python team have been doing
+great work improving support for free-threaded Python across the ecosystem for much of this past year.[^9]


This footnote 9 is missing

rgommers · 2024-12-20T16:14:36Z

apps/labs/posts/libsf-error-state.md

+computing tools and the greatest challenge turned out to be getting things to work on each of them. Several times, just
+as we thought everything was finally working, another quirk would pop up that needed to be addressed.
+
+### Path troubles


Found one more checking off the checkboxes in the PR description: from here on the headings switch from ## to ### - probably not intentional?

steppi · 2024-12-20T17:08:40Z

Thanks @rgommers. I think I've addressed everything except for the inaccuracy regarding why the pip install worked but python dev.py build didn't; much appreciation for your offer to rephrase it.

[BLOG] Add libsf_error_state: SciPy's first shared library

2e4a785

steppi requested review from pavithraes, trallard and rgommers as code owners December 17, 2024 04:05

vercel bot deployed to Preview December 17, 2024 04:09 View deployment

Fix link to n_sphere volume plot

40155cc

vercel bot deployed to Preview December 17, 2024 04:20 View deployment

Add reflections section

5d9914e

vercel bot deployed to Preview December 18, 2024 22:19 View deployment

steppi added 2 commits December 19, 2024 00:46

More updates

4be7421

One more update

d6f6a0b

vercel bot deployed to Preview December 19, 2024 05:56 View deployment

rgommers reviewed Dec 19, 2024

View reviewed changes

Add footnotes

ebfc521

vercel bot deployed to Preview December 19, 2024 16:54 View deployment

Apply easier to implement suggestions from review comments

4c539c7

vercel bot deployed to Preview December 19, 2024 17:42 View deployment

steppi added 2 commits December 19, 2024 12:49

Make mentions of Meson consistent

ef8c630

Move funding acknowledgement to end

5974e5c

vercel bot deployed to Preview December 19, 2024 17:55 View deployment

steppi added 4 commits December 19, 2024 13:02

Some small fixes

949f4a4

Make it clear that we're not aware of anyone doing this first

e2974eb

Change a word choice back

07c5fcb

Make transition to ufuncs less abrupt

0f0e4f2

vercel bot deployed to Preview December 19, 2024 18:24 View deployment

Fix incorrect explanation for why pip was working

1b026bf

vercel bot deployed to Preview December 19, 2024 19:05 View deployment

Fix more inaccuracies

d5dbf8b

vercel bot deployed to Preview December 19, 2024 19:25 View deployment

steppi added 2 commits December 19, 2024 15:47

Further updates

5ef681b

Minor edit

164ce1e

vercel bot deployed to Preview December 19, 2024 20:56 View deployment

Move another footnote to a link in the text

96858ba

vercel bot deployed to Preview December 19, 2024 23:48 View deployment

steppi added 2 commits December 20, 2024 00:03

Another round of small updates

046a070

Add github handle for pv

7d93087

vercel bot deployed to Preview December 20, 2024 05:12 View deployment

rgommers reviewed Dec 20, 2024

View reviewed changes

steppi added 2 commits December 20, 2024 11:36

Respond to review feedback

3ea1696

Remove another unnecessary link

4447651

vercel bot deployed to Preview December 20, 2024 16:45 View deployment


		## NumPy Universal functions

		A NumPy [`ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) represents an arbitrary


		Here's the ufunc [`np.sqrt`](https://numpy.org/doc/stable/reference/generated/numpy.sqrt.html) in action:

		```


		He had a recipe ready to use: defining and using macros which conditionally compiled to the right thing depending on their context.

		```C++

[BLOG] Add libsf_error_state: SciPy's first shared library #892

Are you sure you want to change the base?

[BLOG] Add libsf_error_state: SciPy's first shared library #892

Conversation

steppi commented Dec 17, 2024 • edited by rgommers Loading

Text styling

Non-text contents

vercel bot commented Dec 17, 2024 • edited Loading

rgommers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steppi Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steppi commented Dec 19, 2024 • edited Loading

steppi commented Dec 19, 2024 • edited Loading

steppi commented Dec 19, 2024

steppi commented Dec 19, 2024

rgommers commented Dec 19, 2024

rgommers commented Dec 19, 2024

steppi commented Dec 19, 2024

rgommers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steppi commented Dec 20, 2024

steppi commented Dec 17, 2024 •

edited by rgommers

Loading

vercel bot commented Dec 17, 2024 •

edited

Loading

steppi Dec 19, 2024 •

edited

Loading

steppi commented Dec 19, 2024 •

edited

Loading

steppi commented Dec 19, 2024 •

edited

Loading