Skip to content
This repository has been archived by the owner on Apr 8, 2024. It is now read-only.

Add parameter to shuffle test sets on each node during distributed training #174

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

jfomhover
Copy link
Contributor

@jfomhover jfomhover commented Nov 24, 2021

When using libsvm2bin component, train and test files will be converted together in bin format. It results that for distributed, you have N training files and also N testing files, and both will have to be distributed one per node. But this use case occurs mainy when you use libsvm2bin, although it can be met in other scenarios as well.

This PR implements a parameter in the training module to shuffle the test files one per node, or to share all test files across all nodes. The idea is to use mode n_train_1_test when using common test files, but use n_train_n_test when the number of test files equals number of train files, and should be assigned to each node.

@jfomhover jfomhover added the enhancement New feature or request label Nov 24, 2021
@jfomhover jfomhover added this to the Standardization milestone milestone Nov 24, 2021
@jfomhover jfomhover temporarily deployed to mlops November 24, 2021 00:05 Inactive
@jfomhover jfomhover temporarily deployed to mlops November 24, 2021 00:06 Inactive
@jfomhover jfomhover temporarily deployed to mlops November 24, 2021 00:08 Inactive
@jfomhover jfomhover temporarily deployed to mlops November 24, 2021 00:09 Inactive
@jfomhover jfomhover temporarily deployed to mlops November 24, 2021 00:25 Inactive
@jfomhover jfomhover temporarily deployed to mlops November 24, 2021 00:28 Inactive
@jfomhover jfomhover added the hold This PR/Issue should be put on hold. label Nov 29, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request hold This PR/Issue should be put on hold.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant