Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add Ranking Question to the API #3232

Merged
merged 10 commits into from
Jun 22, 2023
Merged

feat: add Ranking Question to the API #3232

merged 10 commits into from
Jun 22, 2023

Conversation

gabrielmbmb
Copy link
Member

@gabrielmbmb gabrielmbmb commented Jun 20, 2023 •
edited
Loading

Description

This PR adds the required classes to create a Ranking Question and to validate their responses in the API. Additionally, I've refactor the LabelSelectionQuestionSettingsOption class to be named as ValueTextQuestionSettingsOption, as this same class can be used in the Ranking Question.

Closes #3230

Type of change

  • New feature (non-breaking change which adds functionality)
  • Refactor

How Has This Been Tested

Unit tests have been added covering the creation of the Ranking Question as well as the validation of the submitted responses for this kind of question.

Checklist

  • I have merged the original branch into my forked branch
  • I added relevant documentation
  • follows the style guidelines of this project
  • I did a self-review of my code
  • I made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

@gabrielmbmb gabrielmbmb added type: enhancement Indicates new feature requests area: api Indicates that an issue or pull request is related to the Fast API server or REST endpoints labels Jun 20, 2023
@gabrielmbmb gabrielmbmb added this to the v1.12.0 milestone Jun 20, 2023
@codecov
Copy link

codecov bot commented Jun 20, 2023 •
edited
Loading

Codecov Report

Patch coverage: 92.39% and project coverage change: +0.12 🎉

Comparison is base (51751ac) 90.91% compared to head (7bd1837) 91.03%.

❗ Current head 7bd1837 differs from pull request most recent head b8d67e0. Consider uploading reports for the commit b8d67e0 to get more accurate results

Additional details and impacted files 
@@             Coverage Diff             @@
##           develop    #3232      +/-   ##
===========================================
+ Coverage    90.91%   91.03%   +0.12%     
===========================================
  Files          215      218       +3     
  Lines        11304    11548     +244     
===========================================
+ Hits         10277    10513     +236     
- Misses        1027     1035       +8
Flag Coverage Δ
pytest 91.03% <92.39%> (+0.12%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/argilla/__init__.py 87.50% <ø> (+4.16%) ⬆️
src/argilla/server/contexts/datasets.py 96.01% <ø> (ø)
src/argilla/server/seeds.py 0.00% <ø> (ø)
src/argilla/tasks/users/create.py 91.11% <ø> (-4.45%) ⬇️
src/argilla/server/search_engine.py 90.83% <60.00%> (-1.29%) ⬇️
src/argilla/client/apis/datasets.py 91.72% <75.00%> (+1.35%) ⬆️
src/argilla/client/feedback/dataset.py 83.33% <75.00%> (+0.80%) ⬆️
src/argilla/client/feedback/utils.py 78.94% <78.57%> (+7.51%) ⬆️
src/argilla/client/workspaces.py 86.31% <85.86%> (-7.44%) ⬇️
src/argilla/client/sdk/workspaces/api.py 95.12% <96.66%> (+2.81%) ⬆️
... and 31 more

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

frascuchon
frascuchon reviewed Jun 21, 2023
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
frascuchon
frascuchon approved these changes Jun 21, 2023
View reviewed changes
Copy link
Member

@frascuchon frascuchon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's wait for the release branch before merging just to not disturb the release workflow

@frascuchon frascuchon merged commit 06e1f11 into develop Jun 22, 2023
@frascuchon frascuchon deleted the feature/api-ranking-question branch June 22, 2023 17:19
@gabrielmbmb gabrielmbmb mentioned this pull request Jun 23, 2023
Merged
11 tasks
davidberenstein1957 pushed a commit that referenced this pull request Jun 26, 2023
@gabrielmbmb @frascuchon @davidberenstein1957
feat: add Ranking Question to the API (#3232)
d507f86 
# Description

This PR adds the required classes to create a Ranking Question and to
validate their responses in the API. Additionally, I've refactor the
`LabelSelectionQuestionSettingsOption` class to be named as
`ValueTextQuestionSettingsOption`, as this same class can be used in the
Ranking Question.

Closes #3230

**Type of change**

- [x] New feature (non-breaking change which adds functionality)
- [x] Refactor

**How Has This Been Tested**

Unit tests have been added covering the creation of the Ranking Question
as well as the validation of the submitted responses for this kind of
question.

**Checklist**

- [x] I have merged the original branch into my forked branch
- [ ] I added relevant documentation
- [x] follows the style guidelines of this project
- [x] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: Francisco Aranda <francis@argilla.io>
davidberenstein1957 pushed a commit that referenced this pull request Jun 26, 2023
@gabrielmbmb @frascuchon @davidberenstein1957
feat: add Ranking Question to the API (#3232)
05da6ec 
# Description

This PR adds the required classes to create a Ranking Question and to
validate their responses in the API. Additionally, I've refactor the
`LabelSelectionQuestionSettingsOption` class to be named as
`ValueTextQuestionSettingsOption`, as this same class can be used in the
Ranking Question.

Closes #3230

**Type of change**

- [x] New feature (non-breaking change which adds functionality)
- [x] Refactor

**How Has This Been Tested**

Unit tests have been added covering the creation of the Ranking Question
as well as the validation of the submitted responses for this kind of
question.

**Checklist**

- [x] I have merged the original branch into my forked branch
- [ ] I added relevant documentation
- [x] follows the style guidelines of this project
- [x] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: Francisco Aranda <francis@argilla.io>
davidberenstein1957 pushed a commit that referenced this pull request Jun 26, 2023
@gabrielmbmb @frascuchon @davidberenstein1957
feat: add Ranking Question to the API (#3232)
3a10595 
# Description

This PR adds the required classes to create a Ranking Question and to
validate their responses in the API. Additionally, I've refactor the
`LabelSelectionQuestionSettingsOption` class to be named as
`ValueTextQuestionSettingsOption`, as this same class can be used in the
Ranking Question.

Closes #3230

**Type of change**

- [x] New feature (non-breaking change which adds functionality)
- [x] Refactor

**How Has This Been Tested**

Unit tests have been added covering the creation of the Ranking Question
as well as the validation of the submitted responses for this kind of
question.

**Checklist**

- [x] I have merged the original branch into my forked branch
- [ ] I added relevant documentation
- [x] follows the style guidelines of this project
- [x] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: Francisco Aranda <francis@argilla.io>
damianpumar pushed a commit that referenced this pull request Jun 27, 2023
@gabrielmbmb @frascuchon @damianpumar
feat: add Ranking Question to the API (#3232)
116310a 
# Description

This PR adds the required classes to create a Ranking Question and to
validate their responses in the API. Additionally, I've refactor the
`LabelSelectionQuestionSettingsOption` class to be named as
`ValueTextQuestionSettingsOption`, as this same class can be used in the
Ranking Question.

Closes #3230

**Type of change**

- [x] New feature (non-breaking change which adds functionality)
- [x] Refactor

**How Has This Been Tested**

Unit tests have been added covering the creation of the Ranking Question
as well as the validation of the submitted responses for this kind of
question.

**Checklist**

- [x] I have merged the original branch into my forked branch
- [ ] I added relevant documentation
- [x] follows the style guidelines of this project
- [x] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: Francisco Aranda <francis@argilla.io>
@alvarobartt alvarobartt mentioned this pull request Jun 27, 2023
Merged
10 tasks
@frascuchon frascuchon mentioned this pull request Jun 27, 2023
Closed
gabrielmbmb added a commit that referenced this pull request Jun 28, 2023
@alvarobartt @frascuchon @gabrielmbmb
feat: add RankingQuestion in the Python client (#3275)
c1f7aac 
# Description

This PR includes the `RankingQuestion`, introduced in #3232, in the
Python client, to allow users to create datasets with `RankingQuestion`s
and to submit responses for those too.

## Usage

```python
import argilla as rg

rg.init(
    api_url="<ARGILLA_API_URL>",
    api_key="<ARGILLA_API_KEY>,
)

ds = rg.FeedbackDataset(
    fields=[
        rg.TextField(name="prompt-1"),
        rg.TextField(name="prompt-2"),  
    ],
    questions=[
        rg.RankingQuestion(
            name="prompt-ranking",
            description="Rank the prompts from most to least natural.",
            required=True,
            values=["prompt-1", "prompt-2"],
        ),
    ],
)

ds.add_records(
    [
        rg.FeedbackRecord(
            fields={
                "prompt-1": "Explain to a broad audience why banana bread is so fluffy.",
                "prompt-2": "Explain banana banana banana.",
            },
            responses=[
                {
                    "values": {
                        "prompt-ranking": {"value": [{"value": "prompt-1", "rank": 1}, {"value": "prompt-2", "rank": 2}]},
                    },
                    "status": "submitted"
                },
            ],
        ), 
    ]
)

ds.push_to_argilla(name="new-dataset", workspace="new-workspace")
ds = rg.FeedbackDataset.from_argilla(name="new-dataset", workspace="new-workspace")

ds.push_to_huggingface(repo_id="<HUGGINGFACE_REPO_ID>", token="<HUGGINGFACE_TOKEN>")
ds = rg.FeedbackDataset.from_huggingface(repo_id="<HUGGINGFACE_REPO_ID>", token="<HUGGINGFACE_TOKEN>")
```

**Type of change**

- [X] New feature (non-breaking change which adds functionality)

**How Has This Been Tested**

- [X] Added unit tests to cover the new `RankingQuestion`

**Checklist**

- [X] I added relevant documentation
- [X] follows the style guidelines of this project
- [X] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [X] My changes generate no new warnings
- [X] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [X] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: Francisco Aranda <francis@argilla.io>
Co-authored-by: Gabriel Martin <gabriel@argilla.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@frascuchon frascuchon frascuchon approved these changes

Assignees
No one assigned
Labels
area: api Indicates that an issue or pull request is related to the Fast API server or REST endpoints type: enhancement Indicates new feature requests
Projects
None yet
Milestone
v1.12.0
Development

Successfully merging this pull request may close these issues.

[FEATURE] API Ranking Question
2 participants
@gabrielmbmb @frascuchon