Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tensorflow test #378

Merged
merged 14 commits into from
Jun 4, 2020
Merged

tensorflow test #378

merged 14 commits into from
Jun 4, 2020

Conversation

datelier
Copy link
Contributor

Description:

test tensorflow package.

Related Issue:

How Has This Been Tested?:

Environment:

  • Golang Version: 1.14
  • Docker Version: 19.03.5
  • Kubernetes Version: 1.17.3
  • NGT Version: 1.9.1

Types of changes:

  • Bug fix [type/bug]
  • New feature [type/feature]
  • Add tests [type/test]
  • Security related changes [type/security]
  • Add documents [type/documentation]
  • Refactoring [type/refactoring]
  • Update dependencies [type/dependency]
  • Update benchmarks and performances [type/bench]
  • Update CI [type/ci]

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

Checklist:

  • I have read the CONTRIBUTING document.
  • I have checked open Pull Requests for the similar feature or fixes?
  • I have added tests and benchmarks to cover my changes.
  • I have ensured all new and existing tests passed.
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly.

@pull-assistant
Copy link

pull-assistant bot commented May 14, 2020

Score: 0.75

Best reviewed: commit by commit


Optimal code review plan (2 warnings)

Add tensorflow test code

.../tensorflow/tensorflow_test.go 44% changes removed in gofmt

     Add tensorflow option test code

     fix test name

     fix DeepSource issue: Empty string test can be improved

     fix Deepsource issue: Incomplete condition detected

     remove sessionTarget, sessionConfig

     fix DeepSource issue: Empty string test can be improved

     fix test case based on review

gofmt

...erter/tensorflow/tensorflow.go 44% changes removed in delete Closer interf...

     fix golangci-lint issue

     fix golangci-lint issue

     delete Closer interface

     fix DeepSource issue: Function literal can be simplified

     Merge branch 'master' into test/internal/tensorflow

Powered by Pull Assistant. Last update 3d57c14 ... 001fba7. Read the comment docs.

@codecov
Copy link

codecov bot commented May 14, 2020

Codecov Report

Merging #378 into master will increase coverage by 0.94%.
The diff coverage is 86.95%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master    #378      +/-   ##
=========================================
+ Coverage    7.35%   8.29%   +0.94%     
=========================================
  Files         383     372      -11     
  Lines       19664   18847     -817     
=========================================
+ Hits         1446    1564     +118     
+ Misses      17999   17068     -931     
+ Partials      219     215       -4     
Impacted Files Coverage Δ
internal/core/converter/tensorflow/option.go 96.49% <85.71%> (+90.24%) ⬆️
internal/core/converter/tensorflow/tensorflow.go 87.87% <88.88%> (+87.87%) ⬆️
internal/db/storage/blob/s3/option.go 0.00% <0.00%> (-6.67%) ⬇️
internal/config/sidecar.go 0.00% <0.00%> (ø)
internal/config/compress.go 0.00% <0.00%> (ø)
internal/file/watch/watch.go 0.00% <0.00%> (ø)
internal/file/watch/option.go 0.00% <0.00%> (ø)
internal/db/storage/blob/s3/s3.go 0.00% <0.00%> (ø)
pkg/agent/sidecar/config/config.go 0.00% <0.00%> (ø)
pkg/agent/sidecar/usecase/sidecard.go 0.00% <0.00%> (ø)
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48e92aa...001fba7. Read the comment docs.

},
},
{
name: "set value",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "set value",
name: "set success when opts is not nil",

{
name: "set value",
args: args{
opts: &SessionOptions{},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
opts: &SessionOptions{},
opts: new(SessionOptions),

},
want: want{
obj: &T{
options: &SessionOptions{},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
options: &SessionOptions{},
options: new(SessionOptions),

}(),
*/
{
name: "set default",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "set default",
name: "set nothing when tgt is empty",

*/
{
name: "set default",
args: args{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you want to set empty or nil object, you can delete args field.

args: args {
	tgt: ""
}

↑ you can delete `args` fields

}(),
*/
{
name: "return (value, nil)",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "return (value, nil)",
name: "returns (value, nil) when run function returns nil",

checkFunc: defaultCheckFunc,
},
{
name: "run() error",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "run() error",
name: "returns (nil, error) when run function returns `session.Run() error`",

checkFunc: defaultCheckFunc,
},
{
name: "nil tensor error: run() return nil",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "nil tensor error: run() return nil",
name: "returns (nil, error) when
tensors returned by the run function is nil",

},
{
name: "nil tensor error: run() return nil",
args: args{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please delete this field.

checkFunc: defaultCheckFunc,
},
{
name: "nil tensor error: run() return [nil]",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "nil tensor error: run() return [nil]",
name: "returns (nil, error) when
element of tensors returned by the run function is nil",

checkFunc: defaultCheckFunc,
},
{
name: "nil tensor error: run() return nil",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "nil tensor error: run() return nil",
name: "returns (nil, error) when
tensors returned by the run function is nil",

checkFunc: defaultCheckFunc,
},
{
name: "nil tensor error: run() return [nil]",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "nil tensor error: run() return [nil]",
name: "returns (nil, error) when
element of tensors returned by the run function is nil",

checkFunc: defaultCheckFunc,
},
{
name: "failed to cast error: ndim == TwoDim",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "failed to cast error: ndim == TwoDim",
name: "returns (nil, error) when ndim is `TwoDim` and returns error of `ErrFailedToCastTF",

checkFunc: defaultCheckFunc,
},
{
name: "failed to cast error: ndim == ThreeDim",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "failed to cast error: ndim == ThreeDim",
name: "returns (nil, error) when ndim is `ThreeDim ` and returns error of`ErrFailedToCastTF`",

checkFunc: defaultCheckFunc,
},
{
name: "failed to cast error: ndim == default",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "failed to cast error: ndim == default",
name: "returns (nil, error) when ndim is `default ` and returns error of`ErrFailedToCastTF`",

*/
{
name: "return ([], nil): inputs == nil",
args: args{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can delete this field.

}(),
*/
{
name: "return ([], nil): inputs == nil",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "return ([], nil): inputs == nil",
name: "return ([], nil) when inputs is nil and Run function returns ([], nil)",

checkFunc: defaultCheckFunc,
},
{
name: "length error",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "length error",
name: "returns (nil, error) when length of inputs and feeds field are different",

checkFunc: defaultCheckFunc,
},
{
name: "session.Run() error",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "session.Run() error",
name: "returns (nil, error) when Run function returns error",

checkFunc: defaultCheckFunc,
},
{
name: "return ([], nil): inputs == {\"test\"}",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: "return ([], nil): inputs == {\"test\"}",
name: "return ([], nil) when inputs is `{"test"} and Run function returns ([], nil)`",

},
err: nil,
},
checkFunc: defaultCheckFunc,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can delete this line, because this function was set automatically in the tt.Run.

@datelier
Copy link
Contributor Author

/rebase

1 similar comment
@rinx
Copy link
Contributor

rinx commented May 20, 2020

/rebase

@vdaas-ci
Copy link
Collaborator

[REBASE] Rebase triggered by rinx for branch: test/internal/tensorflow

internal/core/converter/tensorflow/tensorflow.go Outdated Show resolved Hide resolved
CloseFunc func() error
}

func (m *mockSession) Run(feeds map[tf.Output]*tf.Tensor, fetches []tf.Output, operations []*Operation) ([]*tf.Tensor, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
line is 127 characters (lll)

options: nil,
graph: nil,
session: &mockSession{
RunFunc: func(feeds map[tf.Output]*tf.Tensor, fetches []tf.Output, operations []*tf.Operation) ([]*tf.Tensor, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
line is 123 characters (lll)

options: nil,
graph: tf.NewGraph(),
session: &mockSession{
RunFunc: func(feeds map[tf.Output]*tf.Tensor, fetches []tf.Output, operations []*tf.Operation) ([]*tf.Tensor, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
line is 123 characters (lll)

options: nil,
graph: tf.NewGraph(),
session: &mockSession{
RunFunc: func(feeds map[tf.Output]*tf.Tensor, fetches []tf.Output, operations []*tf.Operation) ([]*tf.Tensor, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
line is 123 characters (lll)

@@ -56,20 +63,17 @@ const (
ThreeDim
)

var loadFunc = func(exportDir string, tags []string, options *SessionOptions) (*tf.SavedModel, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
loadFunc is a global variable (gochecknoglobals)

want: want{
obj: &T{
feeds: []OutputSpec{
OutputSpec{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
File is not gofmt-ed with -s (gofmt)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@datelier
Could you please format this file?

exportDir: "",
tags: nil,
feeds: []OutputSpec{
OutputSpec{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
File is not gofmt-ed with -s (gofmt)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -109,7 +113,7 @@ func (t *tensorflow) GetVector(inputs ...string) ([]float64, error) {
if err != nil {
return nil, err
}
if tensors == nil || tensors[0] == nil || tensors[0].Value() == nil {
if len(tensors) == 0 || tensors[0] == nil || tensors[0].Value() == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
if statements should only be cuddled with assignments (wsl)

@@ -149,7 +153,7 @@ func (t *tensorflow) GetValue(inputs ...string) (interface{}, error) {
if err != nil {
return nil, err
}
if tensors == nil || tensors[0] == nil {
if len(tensors) == 0 || tensors[0] == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
if statements should only be cuddled with assignments (wsl)

func New(opts ...Option) (TF, error) {
t := new(tensorflow)
for _, opt := range append(defaultOpts, opts...) {
opt(t)
}

if t.options == nil && (len(t.sessionTarget) != 0 || t.sessionConfig != nil) {
if t.options == nil && (t.sessionTarget == "" || t.sessionConfig != nil) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't need nil checking for the string.
you can try on https://play.golang.org/p/_9aagB_YbOM

Suggested change
if t.options == nil && (t.sessionTarget == "" || t.sessionConfig != nil) {
if t.options == nil && t.sessionTarget == "" {

internal/core/converter/tensorflow/tensorflow.go Outdated Show resolved Hide resolved
@kevindiu
Copy link
Contributor

After I run the go test command, the following error occurred.

% go test ./...
go: downloading github.com/cockroachdb/errors v1.2.5-0.20200508151722-0daaf51d1ebe
go: downloading github.com/cockroachdb/sentry-go v0.3.999
# github.com/tensorflow/tensorflow/tensorflow/go
ld: library not found for -ltensorflow
clang: error: linker command failed with exit code 1 (use -v to see invocation)
FAIL	github.com/vdaas/vald/internal/core/converter/tensorflow [build failed]
FAIL

after searching on google i found this link:
https://qiita.com/ysuzuki19/items/044a85efb03f3c290a95

do we need to install the tool before running the go test?

@datelier
Copy link
Contributor Author

@kevindiu
Did you install "TensorFlow C library" before running the go test?
We need to install "TensorFlow C library", so please install this library or do make tensorflow/install.
https://www.tensorflow.org/install/lang_go

if len(tgt) != 0 {
t.sessionTarget = tgt
if tgt != "" {
if t.options == nil {
Copy link
Collaborator

@hlts2 hlts2 May 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's good! 👍

@datelier
Copy link
Contributor Author

/rebase

@vdaas-ci
Copy link
Collaborator

[REBASE] Rebase triggered by datelier for branch: test/internal/tensorflow

@@ -56,25 +63,25 @@ const (
ThreeDim
)

var loadFunc = func(exportDir string, tags []string, options *SessionOptions) (*tf.SavedModel, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
unlambda: replace `func(exportDir string, tags []string, options *SessionOptions) (*tf.SavedModel, error) {

@kevindiu
Copy link
Contributor

@kevindiu
Did you install "TensorFlow C library" before running the go test?
We need to install "TensorFlow C library", so please install this library or do make tensorflow/install.
https://www.tensorflow.org/install/lang_go

I tried to install it but it is failed....

% make tensorflow/install
curl -LO https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.15.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 56.3M  100 56.3M    0     0  12.6M      0  0:00:04  0:00:04 --:--:-- 12.8M
tar -C /usr/local -xzf libtensorflow-cpu-linux-x86_64-1.15.0.tar.gz
./: Can't restore time
./THIRD_PARTY_TF_C_LICENSES: Can't create 'THIRD_PARTY_TF_C_LICENSES'
./LICENSE: Can't create 'LICENSE'
tar: Error exit delayed from previous errors.
make: *** [/usr/local/lib/libtensorflow.so] Error 1

@kevindiu
Copy link
Contributor

Seems the make command is not supported on Mac, I installed the libtensorflow using brew install libtensorflow command and it works. thank you :)

@vankichi
Copy link
Contributor

vankichi commented Jun 2, 2020

/rebase

@vdaas-ci
Copy link
Collaborator

vdaas-ci commented Jun 2, 2020

[REBASE] Rebase triggered by vankichi for branch: test/internal/tensorflow

@vdaas-ci
Copy link
Collaborator

vdaas-ci commented Jun 2, 2020

[REBASE] Failed to rebase.

@vankichi vankichi requested review from hlts2 and kevindiu June 2, 2020 01:58
Signed-off-by: datelier <57349093+datelier@users.noreply.github.com>
@kevindiu
Copy link
Contributor

kevindiu commented Jun 2, 2020

Signed-off-by: datelier <57349093+datelier@users.noreply.github.com>
@kevindiu
Copy link
Contributor

kevindiu commented Jun 3, 2020

LGTM

@kevindiu
Copy link
Contributor

kevindiu commented Jun 3, 2020

/rebase

kevindiu
kevindiu previously approved these changes Jun 3, 2020
@vdaas-ci
Copy link
Collaborator

vdaas-ci commented Jun 3, 2020

[REBASE] Rebase triggered by kevindiu for branch: test/internal/tensorflow

hlts2
hlts2 previously approved these changes Jun 3, 2020
Copy link
Collaborator

@hlts2 hlts2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: datelier <57349093+datelier@users.noreply.github.com>
@datelier datelier dismissed stale reviews from hlts2 and kevindiu via 057e0bd June 3, 2020 05:12
)

var loadFunc = tf.LoadSavedModel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
loadFunc is a global variable (gochecknoglobals)

Copy link
Collaborator

@hlts2 hlts2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kevindiu
Copy link
Contributor

kevindiu commented Jun 3, 2020

LGTM

@kevindiu kevindiu requested a review from vankichi June 3, 2020 06:20
Copy link
Contributor

@vankichi vankichi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vankichi vankichi added status/author-review and removed team/set SET team labels Jun 4, 2020
@vankichi vankichi requested review from kpango and rinx June 4, 2020 03:35
@vankichi
Copy link
Contributor

vankichi commented Jun 4, 2020

@kpango @rinx
Could you review this PR by 6/9 ? 🙏

Copy link
Collaborator

@kpango kpango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kpango kpango merged commit 6034bb8 into master Jun 4, 2020
@kpango kpango deleted the test/internal/tensorflow branch June 4, 2020 04:52
@github-actions github-actions bot added the team/set SET team label Jun 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants