Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RPC] Replace timestamp with counter #7389

Merged
merged 4 commits into from
Feb 5, 2021
Merged

Conversation

comaniac
Copy link
Contributor

@comaniac comaniac commented Feb 1, 2021

When tuning workload using AutoTVM, we may encounter an exception saying < not supported between instances of function and function. The reason is we currently put RPC requests in the format of (-priority, time.time(), callback). When two requests with the same priority and time.time(), then heap pop throws the error of trying to compare two callbacks. Although it seems rare to have the same timestamp for two requests, one of my colleagues encounters an exception repeatedly when tuning a task using AutoTVM on V100 GPU. As a result, I try to use a counter to replace the timestamp so that we guarantee there's no duplication. Not sure if we need thread locking here to make sure thread safe, tho.

cc @tqchen @FrozenGene

python/tvm/rpc/tracker.py Outdated Show resolved Hide resolved
@comaniac
Copy link
Contributor Author

comaniac commented Feb 4, 2021

cc @tqchen to take a final look and merge.

@tqchen tqchen merged commit f1b9663 into apache:main Feb 5, 2021
@comaniac comaniac deleted the fix_rpc_request branch February 5, 2021 00:13
alexwong pushed a commit to alexwong/tvm that referenced this pull request Feb 11, 2021
Lokiiiiii pushed a commit to Lokiiiiii/tvm that referenced this pull request Mar 2, 2021
trevor-m pushed a commit to neo-ai/tvm that referenced this pull request Mar 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants