Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Re-org #690

Merged
merged 17 commits into from
Oct 23, 2020
Merged

API Re-org #690

merged 17 commits into from
Oct 23, 2020

Conversation

asutula
Copy link
Member

@asutula asutula commented Oct 22, 2020

71 of those files changed are deleted files ;)

The deletions are a lot of removing API that isn't primary to the current direction and focus of Powergate.

All other changes are refactoring of the API to make it easier to understand and use. This means removing the concept of and word "ffs" from the API and organizing the API around functionality/features instead of backend implementation concepts. My main driving thought was to make the primary interaction points of the API (getting, setting, applying configs, etc) top-level and easily accessible, while nesting secondary functionality (querying for job status, wallet balance, etc) one level down in appropriately named sub modules.

Here is the entire Powergate API now, with the comments denoting the top level and sub modules. This hierarchy is currently used as a road map and applied directly to the Go client and CLI structure. To reiterate: All changes in this PR, from RPC service implementation to client to CLI, are just refactoring to match this API design.

service PowergateService {
  // Top level
  rpc BuildInfo(BuildInfoRequest) returns (BuildInfoResponse) {}
  rpc ID(IDRequest) returns (IDResponse) {}
  rpc DefaultStorageConfig(DefaultStorageConfigRequest) returns (DefaultStorageConfigResponse) {}
  rpc SetDefaultStorageConfig(SetDefaultStorageConfigRequest) returns (SetDefaultStorageConfigResponse) {}
  rpc Stage(stream StageRequest) returns (StageResponse) {}
  rpc ApplyStorageConfig(ApplyStorageConfigRequest) returns (ApplyStorageConfigResponse) {}
  rpc ReplaceData(ReplaceDataRequest) returns (ReplaceDataResponse) {}
  rpc Get(GetRequest) returns (stream GetResponse) {}
  rpc Remove(RemoveRequest) returns (RemoveResponse) {}
  rpc WatchLogs(WatchLogsRequest) returns (stream WatchLogsResponse){}
  rpc CidInfo(CidInfoRequest) returns (CidInfoResponse) {}

  // Wallet
  rpc Balance(BalanceRequest) returns (BalanceResponse) {}
  rpc NewAddr(NewAddrRequest) returns (NewAddrResponse) {}
  rpc Addrs(AddrsRequest) returns (AddrsResponse) {}
  rpc SendFil(SendFilRequest) returns (SendFilResponse) {}
  rpc SignMessage(SignMessageRequest) returns (SignMessageResponse) {}
  rpc VerifyMessage(VerifyMessageRequest) returns (VerifyMessageResponse) {}

  // Storage Jobs
  rpc StorageJob(StorageJobRequest) returns (StorageJobResponse) {}
  rpc StorageConfigForJob(StorageConfigForJobRequest) returns (StorageConfigForJobResponse) {}
  rpc QueuedStorageJobs(QueuedStorageJobsRequest) returns (QueuedStorageJobsResponse) {}
  rpc ExecutingStorageJobs(ExecutingStorageJobsRequest) returns (ExecutingStorageJobsResponse) {}
  rpc LatestFinalStorageJobs(LatestFinalStorageJobsRequest) returns (LatestFinalStorageJobsResponse) {}
  rpc LatestSuccessfulStorageJobs(LatestSuccessfulStorageJobsRequest) returns (LatestSuccessfulStorageJobsResponse) {}
  rpc StorageJobsSummary(StorageJobsSummaryRequest) returns (StorageJobsSummaryResponse) {}
  rpc WatchStorageJobs(WatchStorageJobsRequest) returns (stream WatchStorageJobsResponse) {}
  rpc CancelStorageJob(CancelStorageJobRequest) returns (CancelStorageJobResponse) {}
  
  // Deals
  rpc ListStorageDealRecords(ListStorageDealRecordsRequest) returns (ListStorageDealRecordsResponse) {}
  rpc ListRetrievalDealRecords(ListRetrievalDealRecordsRequest) returns (ListRetrievalDealRecordsResponse) {}
}

Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
@asutula asutula self-assigned this Oct 22, 2020
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
@asutula asutula requested a review from jsign October 23, 2020 17:10
@asutula asutula marked this pull request as ready for review October 23, 2020 17:10
@asutula
Copy link
Member Author

asutula commented Oct 23, 2020

Almost all of the API is implemented using FFS on the back end... therefore providing the POW_TOKEN is just a global thing now and should be done always.

@asutula
Copy link
Member Author

asutula commented Oct 23, 2020

Also for reference, here is the full admin API. We can provide more hierarchical structure in the admin client and CLI at some point, but for now they are just implemented as a flat Admin sub module of the client and CLI:

service PowergateAdminService {
  // Wallet
  rpc NewAddress(NewAddressRequest) returns (NewAddressResponse) {}
  rpc ListAddresses(ListAddressesRequest) returns (ListAddressesResponse) {}
  rpc SendFil(SendFilRequest) returns (SendFilResponse) {}

  // Instances
  rpc CreateStorageProfile(CreateStorageProfileRequest) returns (CreateStorageProfileResponse) {}
  rpc ListStorageProfiles(ListStorageProfilesRequest) returns (ListStorageProfilesResponse) {}

  // Jobs
  rpc QueuedStorageJobs(QueuedStorageJobsRequest) returns (QueuedStorageJobsResponse) {}
  rpc ExecutingStorageJobs(ExecutingStorageJobsRequest) returns (ExecutingStorageJobsResponse) {}
  rpc LatestFinalStorageJobs(LatestFinalStorageJobsRequest) returns (LatestFinalStorageJobsResponse) {}
  rpc LatestSuccessfulStorageJobs(LatestSuccessfulStorageJobsRequest) returns (LatestSuccessfulStorageJobsResponse) {}
  rpc StorageJobsSummary(StorageJobsSummaryRequest) returns (StorageJobsSummaryResponse) {}
}

Signed-off-by: Aaron Sutula <hi@asutula.com>
Signed-off-by: Aaron Sutula <hi@asutula.com>
Copy link
Contributor

@jsign jsign left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lots of cleaning, moving and renaming

Eager to see how it feels using all these changes!

@asutula asutula merged commit f9f5683 into asutula/deal-insight Oct 23, 2020
@asutula asutula deleted the asutula/api-reorg branch October 23, 2020 18:45
asutula added a commit that referenced this pull request Nov 4, 2020
* represent deal states with enum

Signed-off-by: Aaron Sutula <hi@asutula.com>

* bubble up deal status changes

Signed-off-by: Aaron Sutula <hi@asutula.com>

* lock access to cache

Signed-off-by: Aaron Sutula <hi@asutula.com>

* typo and indent fix

Signed-off-by: Aaron Sutula <hi@asutula.com>

* pr feedback, send StorageDealInfo updates

Signed-off-by: Aaron Sutula <hi@asutula.com>

* little cleanup for pr feedback

Signed-off-by: Aaron Sutula <hi@asutula.com>

* move job monitoring inside job store, extend api to client and cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better chan handling and locking

Signed-off-by: Aaron Sutula <hi@asutula.com>

* table output improvements

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fixed table output

Signed-off-by: Aaron Sutula <hi@asutula.com>

* user protojson

Signed-off-by: Aaron Sutula <hi@asutula.com>

* seeing how raw json output feels

Signed-off-by: Aaron Sutula <hi@asutula.com>

* raw grpc and json for net client and cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* caches galore

Signed-off-by: Aaron Sutula <hi@asutula.com>

* expose data in api, tests

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove func options, add api to sched and ffs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* admin service, client code

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix some tests

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove redundant token flag, ffs client and cl for job queries

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better method naming

Signed-off-by: Aaron Sutula <hi@asutula.com>

* all sorts of work

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fixed some array building

Signed-off-by: Aaron Sutula <hi@asutula.com>

* new general service

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update lotus to v0.10.0

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* admin token

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* revert cidinfo name refactor

Signed-off-by: Aaron Sutula <hi@asutula.com>

* include tests

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* docs update

Signed-off-by: Aaron Sutula <hi@asutula.com>

* cleaning up a few things

Signed-off-by: Aaron Sutula <hi@asutula.com>

* revert net changes for now

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix net test

Signed-off-by: Aaron Sutula <hi@asutula.com>

* organizing services

Signed-off-by: Aaron Sutula <hi@asutula.com>

* clean up admin auth

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix some tests, admin cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* use wallet module in powergate service

Signed-off-by: Aaron Sutula <hi@asutula.com>

* finish merge

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update docs, couple nits

Signed-off-by: Aaron Sutula <hi@asutula.com>

* clean up some nits, use new powergate service for balance, client/cli for storage config by job id

Signed-off-by: Aaron Sutula <hi@asutula.com>

* docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove locks

Signed-off-by: Aaron Sutula <hi@asutula.com>

* correct locking

Signed-off-by: Aaron Sutula <hi@asutula.com>

* pr feedback

Signed-off-by: Aaron Sutula <hi@asutula.com>

* just continue in case of job query error

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Client/CLI simplifications and improvements (#686)

* use basic types and return grpc types

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update cli, output json most of the time, better json

Signed-off-by: Aaron Sutula <hi@asutula.com>

* put back SendFil

Signed-off-by: Aaron Sutula <hi@asutula.com>

* docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Storage profiles with tokens (#688)

* storage profiles with tokens

Signed-off-by: Aaron Sutula <hi@asutula.com>

* comment export

Signed-off-by: Aaron Sutula <hi@asutula.com>

* return AuthEntry from Create

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Repurpose CidInfo (#689)

* rename cidinfo to storageinfo

Signed-off-by: Aaron Sutula <hi@asutula.com>

* rename ciddata to cidinfo

Signed-off-by: Aaron Sutula <hi@asutula.com>

* docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* API Re-org (#690)

* move buildinfo service to new foundation

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove health module, rpc, client, cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove indexes and reputation apis

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove net module and api

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix proto formatting

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove pay channel code

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove wallet module rpc, move wallet methods to admin api

Signed-off-by: Aaron Sutula <hi@asutula.com>

* add wallet admin apis to client and cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* set wm

Signed-off-by: Aaron Sutula <hi@asutula.com>

* move ffs rpc to powergate proto, update client

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update runner

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* future proofing some api names

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Make config commands top level

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove extra line

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove some ffs wording

Signed-off-by: Aaron Sutula <hi@asutula.com>

* more ffs word removal

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update pow docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix auth context

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update readme docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* api categories for config and data

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better admin client and cli structure

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update wallet type default

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix up client tests

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix formatting

Signed-off-by: Aaron Sutula <hi@asutula.com>

* cleaning up wallet api names

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better naming for list methods

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Better names for client jobs methods

Signed-off-by: Aaron Sutula <hi@asutula.com>

* profile id naming

Signed-off-by: Aaron Sutula <hi@asutula.com>

* clean up job id name in api

Signed-off-by: Aaron Sutula <hi@asutula.com>

* cleaning up some other api names

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better name for storage profile id

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better client options for admin storage jobs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* clean up logging

Signed-off-by: Aaron Sutula <hi@asutula.com>

* introducing strings to some fil amounts

Signed-off-by: Aaron Sutula <hi@asutula.com>

* change balance and sendfil over to big int

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better proto name for address type

Signed-off-by: Aaron Sutula <hi@asutula.com>

Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants