Inline project meta

[dmpetrov]

Description

To increase scripts reusability, it would be great to have all project meta in one place.

An alternative approach - using https://peps.python.org/pep-0723/

# __pyproject__
"""
[project]
requires-python = ">=3.10"
dependencies = [
    "moviepy < 2.0"
]

[tool.datachain.inputs]
input_bucket = "gs://mybucket"
input_dir = "examples/videos/"

[tool.datachain.params]
min_length_sec = 1
cache_input = False

[tool.datachain.attachments]
yolo_model="gs://mybucket/share/yolov8m_holding_v4.pt"


[tool.datachain.outputs]
result_dataset = "ds://res"
result_dir = "{input_bucket}/temp"
"""

import io
from datachain import DataChain, File, C, inline
...

bucket = inline.get_input("input_bucket")
dir = inline.get_input("input_dir")
result_dataset = inline.get_output("result_dataset")
duration_limit = inline.get_params("min_length_sec")

[ilongin]

On the first look, I like this approach over script metadata.
We can also add ability to upload this config file separately in Studio UI when running scripts, as users could have it saved somewhere and reuse it on every script run.
So user would have 3 options:

1. Inline config in script itself as in example
2. Upload config separately
3. Add config values manually as it's done now -> this should be transformed again to config file / format in the backend.

WDYT?

[dmpetrov]

We should prioritize a unified approach that works in both - UI as well as CLI. User should be able to copy-past from CLI to SaaS or opposite without even thinking about this 🙂

So, it's ok to have a customization in UI (as we have it now) but I'd try to deprioritize it and potentially get rid ot it completely.

[dmpetrov]

uv run myscript.py natively supports the comment way - PEP 723. It looks really good!

https://www.linkedin.com/posts/julienhuraultanalytics_uv-is-so-cool-i-just-discovered-how-to-activity-7282422214799286272-WV5N

[shcheklein]

that looks cool! :)

[dmpetrov]

I've tested this - works like a charm.

More details:

• 👍 Installing packages works fast as usual with uv
• 👎 Preparing packages/"compiling" takes a lot of time in the firs run.
• 👍 Caching - Following runs do not require installation or preporation.

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "pandas < 2.1.0"
# ]
# ///

import sys
import pandas as pd

print(f"Python version: {sys.version_info}")
print(f"Pandas version: {pd.__version__}")

First run:

$ time uv run test_inline.py
Reading inline script metadata from `test_inline.py`
Installed 6 packages in 114ms
Python version: sys.version_info(major=3, minor=13, micro=1, releaselevel='final', serial=0)
Pandas version: 2.2.3

real    0m17.804s
user    0m2.084s
sys     0m1.277s

Next run:

$ time uv run test_inline.py
Reading inline script metadata from `test_inline.py`
Python version: sys.version_info(major=3, minor=13, micro=1, releaselevel='final', serial=0)
Pandas version: 2.2.3

real    0m0.443s
user    0m0.344s
sys     0m0.087s