Running pytest on Databricks might fail if filesystem is read-only

[astrojuanlu]

Description

A user reported the other day that they were getting an OSError: [Errno 95] Operation not supported error mentioning the __pycache__ directory after running the tests on Databricks:

(source https://linen-slack.kedro.org/t/15886807/team-i-am-trying-to-run-pytest-on-databricks-and-getting-thi#d700145d-a167-4215-8dda-fedb15c5545a)

The Databricks documentation on running unit tests https://docs.databricks.com/en/notebooks/testing.html#run-unit-tests specifies this line:

...
# Skip writing pyc files on a readonly filesystem.
sys.dont_write_bytecode = True

We could maybe add this to our docs, or put it in our starters.

Alternatively, we might decide to not do anything, because this is hardly our fault, and we'd have to keep up with whatever Databricks does.

[stichbury]

It's unfortunate that the error is not documented in the Slack message except as an image since it would itself be sufficient to index it so anyone searching google to find an answer would hit the linen archive and get a solution.

Let's do it here so it's clear (and @astrojuanlu has already done this above) and this will count as our documentation of this very specific issue.

Issue summary

You may get an OSError: [Errno 95] Operation not supported error mentioning the __pycache__ directory after running the pytest tests on Databricks:

Resolution

The Databricks documentation on running unit tests https://docs.databricks.com/en/notebooks/testing.html#run-unit-tests specifies this line:

# Skip writing pyc files on a readonly filesystem.
sys.dont_write_bytecode = True