-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-20544][SPARKR] R wrapper for input_file_name #17818
Conversation
Test build #76344 has finished for PR 17818 at commit
|
Test build #76357 has finished for PR 17818 at commit
|
8457f01
to
2dd17dc
Compare
Test build #76358 has finished for PR 17818 at commit
|
Test build #76359 has finished for PR 17818 at commit
|
hmm, not clear why AppVeyor failed. you could trigger it again by closing and re-opening this PR |
R/pkg/R/functions.R
Outdated
|
||
#' input_file_name | ||
#' | ||
#' Creates a string column for the file name of the current Spark task. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I actually find this description in Scala API quite a bit confusing - what is "Spark task" and how it has "file name"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about the new one?
path <- tempfile(pattern = "input_file_name_test", fileext = ".txt") | ||
write.table(iris[1:50, ], path, row.names = FALSE, col.names = FALSE) | ||
|
||
df <- read.text(path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does it work with df <- read.json(jsonPath)
?
if yes, consider adding to the test for column functions
(again, this is regarding: #17817 to consolidate/skip tests)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should work with any file input as far as I remember. I'd skip collecting but there have been some issues with PySpark in the past.
Look I'll have to rebase it anyway but thank you so much for the hint. I've been meaning to ask if there is some equivalent of Jenkins helpers. |
Test build #76375 has finished for PR 17818 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
R/pkg/R/functions.R
Outdated
#' | ||
#' @rdname input_file_name | ||
#' @name input_file_name | ||
#' @aliases input_file_name,missing-method |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually, could you add @family normal_funcs
here? I missed this earlier and in the other PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
Test build #76397 has finished for PR 17818 at commit
|
|
||
#' input_file_name | ||
#' | ||
#' Creates a string column with the input file name for a given row |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this actually makes a lot more sense...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
merged to master |
What changes were proposed in this pull request?
Adds wrapper for
o.a.s.sql.functions.input_file_name
How was this patch tested?
Existing unit tests, additional unit tests,
check-cran.sh
.