Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Finish SQL functions refactoring #543

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Finish SQL functions refactoring #543

wants to merge 1 commit into from
+1,146 −250

Conversation

dreadatour
Copy link
Contributor

@dreadatour dreadatour commented Oct 28, 2024 •
edited
Loading

  • Move all functions from datachain.sql.functions to datachain.lib.func (hide sql from user).
  • Wrap all functions with custom wrapper — this allows us to know the type of output signal
  • Add more docstrings for functions

@dreadatour dreadatour self-assigned this Oct 28, 2024
@dreadatour dreadatour marked this pull request as draft October 28, 2024 12:44
@cloudflare-workers-and-pages Cloudflare Workers and Pages
Copy link

cloudflare-workers-and-pages bot commented Oct 28, 2024 •
edited
Loading

Deploying datachain-documentation with  Cloudflare Pages  Cloudflare Pages

Latest commit: 21a9923
Status: ✅  Deploy successful!
Preview URL: https://0991e841.datachain-documentation.pages.dev
Branch Preview URL: https://525-sql-functions.datachain-documentation.pages.dev

View logs

@dreadatour dreadatour linked an issue Oct 28, 2024 that may be closed by this pull request
Finish SQL functions refactoring #525
Open
@dreadatour dreadatour force-pushed the 525-sql-functions branch 4 times, most recently from 04c16a5 to 00cdb0d Compare October 29, 2024 01:58
@codecov Codecov
Copy link

codecov bot commented Oct 29, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 80.05952% with 67 lines in your changes missing coverage. Please review.

Project coverage is 87.08%. Comparing base (b62d091) to head (21a9923).

Files with missing lines Patch % Lines
src/datachain/lib/func/func.py 64.03% 35 Missing and 6 partials ⚠️
src/datachain/lib/func/string.py 74.28% 6 Missing and 3 partials ⚠️
src/datachain/lib/dc.py 78.94% 4 Missing and 4 partials ⚠️
src/datachain/lib/func/array.py 85.00% 6 Missing ⚠️
src/datachain/cli.py 0.00% 1 Missing ⚠️
src/datachain/client/fsspec.py 83.33% 0 Missing and 1 partial ⚠️
src/datachain/lib/func/inner/base.py 87.50% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #543      +/-   ##
==========================================
- Coverage   87.41%   87.08%   -0.33%     
==========================================
  Files         106      111       +5     
  Lines       10279    10511     +232     
  Branches     1377     1423      +46     
==========================================
+ Hits         8985     9154     +169     
- Misses        953     1005      +52     
- Partials      341      352      +11
Flag Coverage Δ
datachain 87.03% <80.05%> (-0.33%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dreadatour dreadatour force-pushed the 525-sql-functions branch 6 times, most recently from 83cbb49 to 26fb69c Compare October 30, 2024 03:04
@dreadatour dreadatour requested a review from a team November 17, 2024 15:36
@dreadatour dreadatour marked this pull request as ready for review November 17, 2024 15:36
mattseddon
mattseddon approved these changes Nov 17, 2024
View reviewed changes
examples/computer_vision/openimage-detect.py
@@ -4,7 +4,7 @@
from pydantic import BaseModel

from datachain import C, DataChain, File
from datachain.sql.functions import path
from datachain.lib.func import path
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Q] Do we want to import func from datachain instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

** same for all examples - I don't think it matters which approach we take but consistency between examples would be good

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right. lib needs to be deprecated. User should import like from datachain.func import path

src/datachain/data_storage/warehouse.py
@@ -20,8 +20,8 @@
from datachain.data_storage.schema import convert_rows_custom_column_types
from datachain.data_storage.serializer import Serializable
from datachain.dataset import DatasetRecord, StorageURI
from datachain.lib.func.inner import path as pathfunc
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Q] What is the difference between datachain.lib.func & datachain.lib.func.inner? When would you use one and not the other?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dmpetrov dmpetrov dmpetrov left review comments

@mattseddon mattseddon mattseddon approved these changes

Assignees

@dreadatour dreadatour

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Finish SQL functions refactoring
3 participants
@dreadatour @dmpetrov @mattseddon