Skip to content

Commit

Permalink
Deal Insight + New API Foundation (#659)
Browse files Browse the repository at this point in the history 
* represent deal states with enum

Signed-off-by: Aaron Sutula <hi@asutula.com>

* bubble up deal status changes

Signed-off-by: Aaron Sutula <hi@asutula.com>

* lock access to cache

Signed-off-by: Aaron Sutula <hi@asutula.com>

* typo and indent fix

Signed-off-by: Aaron Sutula <hi@asutula.com>

* pr feedback, send StorageDealInfo updates

Signed-off-by: Aaron Sutula <hi@asutula.com>

* little cleanup for pr feedback

Signed-off-by: Aaron Sutula <hi@asutula.com>

* move job monitoring inside job store, extend api to client and cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better chan handling and locking

Signed-off-by: Aaron Sutula <hi@asutula.com>

* table output improvements

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fixed table output

Signed-off-by: Aaron Sutula <hi@asutula.com>

* user protojson

Signed-off-by: Aaron Sutula <hi@asutula.com>

* seeing how raw json output feels

Signed-off-by: Aaron Sutula <hi@asutula.com>

* raw grpc and json for net client and cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* caches galore

Signed-off-by: Aaron Sutula <hi@asutula.com>

* expose data in api, tests

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove func options, add api to sched and ffs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* admin service, client code

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix some tests

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove redundant token flag, ffs client and cl for job queries

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better method naming

Signed-off-by: Aaron Sutula <hi@asutula.com>

* all sorts of work

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fixed some array building

Signed-off-by: Aaron Sutula <hi@asutula.com>

* new general service

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update lotus to v0.10.0

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* admin token

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* revert cidinfo name refactor

Signed-off-by: Aaron Sutula <hi@asutula.com>

* include tests

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* docs update

Signed-off-by: Aaron Sutula <hi@asutula.com>

* cleaning up a few things

Signed-off-by: Aaron Sutula <hi@asutula.com>

* revert net changes for now

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix net test

Signed-off-by: Aaron Sutula <hi@asutula.com>

* organizing services

Signed-off-by: Aaron Sutula <hi@asutula.com>

* clean up admin auth

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix some tests, admin cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* use wallet module in powergate service

Signed-off-by: Aaron Sutula <hi@asutula.com>

* finish merge

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update docs, couple nits

Signed-off-by: Aaron Sutula <hi@asutula.com>

* clean up some nits, use new powergate service for balance, client/cli for storage config by job id

Signed-off-by: Aaron Sutula <hi@asutula.com>

* docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove locks

Signed-off-by: Aaron Sutula <hi@asutula.com>

* correct locking

Signed-off-by: Aaron Sutula <hi@asutula.com>

* pr feedback

Signed-off-by: Aaron Sutula <hi@asutula.com>

* just continue in case of job query error

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Client/CLI simplifications and improvements (#686)

* use basic types and return grpc types

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update cli, output json most of the time, better json

Signed-off-by: Aaron Sutula <hi@asutula.com>

* put back SendFil

Signed-off-by: Aaron Sutula <hi@asutula.com>

* docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Storage profiles with tokens (#688)

* storage profiles with tokens

Signed-off-by: Aaron Sutula <hi@asutula.com>

* comment export

Signed-off-by: Aaron Sutula <hi@asutula.com>

* return AuthEntry from Create

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Repurpose CidInfo (#689)

* rename cidinfo to storageinfo

Signed-off-by: Aaron Sutula <hi@asutula.com>

* rename ciddata to cidinfo

Signed-off-by: Aaron Sutula <hi@asutula.com>

* docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* API Re-org (#690)

* move buildinfo service to new foundation

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove health module, rpc, client, cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove indexes and reputation apis

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove net module and api

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix proto formatting

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove pay channel code

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove wallet module rpc, move wallet methods to admin api

Signed-off-by: Aaron Sutula <hi@asutula.com>

* add wallet admin apis to client and cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* set wm

Signed-off-by: Aaron Sutula <hi@asutula.com>

* move ffs rpc to powergate proto, update client

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update runner

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update cli

Signed-off-by: Aaron Sutula <hi@asutula.com>

* future proofing some api names

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Make config commands top level

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove extra line

Signed-off-by: Aaron Sutula <hi@asutula.com>

* remove some ffs wording

Signed-off-by: Aaron Sutula <hi@asutula.com>

* more ffs word removal

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update pow docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix auth context

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update readme docs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* api categories for config and data

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better admin client and cli structure

Signed-off-by: Aaron Sutula <hi@asutula.com>

* update wallet type default

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix up client tests

Signed-off-by: Aaron Sutula <hi@asutula.com>

* fix formatting

Signed-off-by: Aaron Sutula <hi@asutula.com>

* cleaning up wallet api names

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better naming for list methods

Signed-off-by: Aaron Sutula <hi@asutula.com>

* Better names for client jobs methods

Signed-off-by: Aaron Sutula <hi@asutula.com>

* profile id naming

Signed-off-by: Aaron Sutula <hi@asutula.com>

* clean up job id name in api

Signed-off-by: Aaron Sutula <hi@asutula.com>

* cleaning up some other api names

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better name for storage profile id

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better client options for admin storage jobs

Signed-off-by: Aaron Sutula <hi@asutula.com>

* clean up logging

Signed-off-by: Aaron Sutula <hi@asutula.com>

* introducing strings to some fil amounts

Signed-off-by: Aaron Sutula <hi@asutula.com>

* change balance and sendfil over to big int

Signed-off-by: Aaron Sutula <hi@asutula.com>

* better proto name for address type

Signed-off-by: Aaron Sutula <hi@asutula.com>

Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
  • Loading branch information
@asutula @jsign
asutula and jsign committed Nov 4, 2020
1 parent 4083325 commit 363d03b
Show file tree
Hide file tree
Showing 280 changed files with 16,104 additions and 20,720 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ jobs:
replace-with: ""
- name: Generate JS gRPC bindings
run: |
./scripts/gen-js-protos.sh ${{steps.makeversion.outputs.replaced}} . ./js-grpc
./scripts/gen-js-protos.sh ${{steps.makeversion.outputs.replaced}} . ./js-grpc
- name: Publish JS gRPC bindings
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_AUTH_TOKEN }}
Expand Down
103 changes: 66 additions & 37 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ Want to know more about this Powergate module? Check out the [FFS design documen

### 💫 API + CLI

Powergate expose modules functionalities through gRPC endpoints.
You can explore our `.proto` files to generate your clients, or take advange of a ready-to-use Powergate Go and [JS client](https://github.com/textileio/js-powergate-client). 🙌
Powergate exposes an API built from the various modules through gRPC endpoints.
You can explore our [`.proto` files](https://github.com/textileio/powergate/proto) to generate your clients, or take advange of a ready-to-use Powergate Go and [JS client](https://github.com/textileio/js-powergate-client). 🙌

We have a CLI that supports most of Powergate features.

Expand All @@ -75,27 +75,29 @@ The binary will be placed automatically in `$GOPATH/bin` which in general is in
You can read the [generated CLI docs](https://github.com/textileio/powergate/blob/master/cli-docs/pow/pow.md) in this repo, or run `pow` with the `--help` flag to see the available commands:

```
$ pow --help
A client for storage and retreival of powergate data
Usage:
pow [flags]
pow [command]
Available Commands:
asks Provides commands to view asks data
faults Provides commands to view faults data
ffs Provides commands to manage ffs
health Display the node health status
help Help about any command
miners Provides commands to view miners data
net Provides commands related to peers and network
reputation Provides commands to view miner reputation data
wallet Provides commands about filecoin wallets
admin Provides admin commands
config Provides commands to interact with cid storage configs
data Provides commands to interact with general data APIs
deals Provides commands to view Filecoin deal information
help Help about any command
id Returns the storage profile id
storage-jobs Provides commands to query for storage jobs in various states
version Display version information for pow and the connected server
wallet Provides commands about filecoin wallets
Flags:
-h, --help help for pow
--serverAddress string address of the powergate service api (default "127.0.0.1:5002")
-t, --token string storage profile auth token
-v, --version display version information for pow and the connected server
Use "pow [command] --help" for more information about a command.
```
Expand Down Expand Up @@ -136,23 +138,39 @@ make install-powd
```
You can run the `-h` flag to see the configurable flags:
```bash
$ powd -h
$ powd -h
Usage of powd:
--autocreatemasteraddr Automatically creates & funds a master address if none is provided
--debug Enable debug log level in all loggers.
--devnet Indicate that will be running on an ephemeral devnet. --repopath will be autocleaned on exit.
--ffsusemasteraddr Use the master address as the initial address for all new FFS instances instead of creating a new unique addess for each new FFS instance.
--gatewayhostaddr string Gateway host listening address (default "0.0.0.0:7000")
--grpchostaddr string gRPC host listening address. (default "/ip4/0.0.0.0/tcp/5002")
--grpcwebproxyaddr string gRPC webproxy listening address. (default "0.0.0.0:6002")
--ipfsapiaddr string IPFS API endpoint multiaddress. (Optional, only needed if FFS is used) (default "/ip4/127.0.0.1/tcp/5001")
--lotushost string Lotus client API endpoint multiaddress. (default "/ip4/127.0.0.1/tcp/1234")
--lotusmasteraddr string Existing wallet address in Lotus to be used as source of funding for new FFS instances. (Optional)
--lotustoken string Lotus API authorization token. This flag or --lotustoken file are mandatory.
--lotustokenfile string Path of a file that contains the Lotus API authorization token.
--maxminddbfolder string Path of the folder containing GeoLite2-City.mmdb (default ".")
--repopath string Path of the repository where Powergate state will be saved. (default "~/.powergate")
--walletinitialfund int FFS initial funding transaction amount in attoFIL received by --lotusmasteraddr. (if set) (default 4000000000000000)
--askindexmaxparallel string Max parallel query ask to execute while updating index (default "3")
--askindexqueryasktimeout string Timeout in seconds for a query ask (default "15")
--askindexrefreshinterval string Refresh interval measured in minutes (default "60")
--askindexrefreshonstart If true it will refresh the index on start
--autocreatemasteraddr Automatically creates & funds a master address if none is provided.
--dealwatchpollduration string Poll interval in seconds used by Deals Module watch to detect state changes (default "900")
--debug Enable debug log level in all loggers.
--devnet Indicate that will be running on an ephemeral devnet. --repopath will be autocleaned on exit.
--disableindices Disable all indices updates, useful to help Lotus syncing process
--disablenoncompliantapis Disable APIs that may not easily comply with US law
--ffsadmintoken string FFS admin token for authorized APIs. If empty, the APIs will be open to the public.
--ffsdealfinalitytimeout string Deadline in minutes in which a deal must prove liveness changing status before considered abandoned (default "4320")
--ffsminerselector string Miner selector to be used by FFS: 'sr2', 'reputation' (default "sr2")
--ffsminerselectorparams string Miner selector configuration parameter, depends on --ffsminerselector (default "https://raw.githubusercontent.com/filecoin-project/slingshot/master/miners.json")
--ffsminimumpiecesize string Minimum piece size in bytes allowed to be stored in Filecoin (default "67108864")
--ffsschedmaxparallel string Maximum amount of Jobs executed in parallel (default "1000")
--ffsusemasteraddr Use the master address as the initial address for all new FFS instances instead of creating a new unique addess for each new FFS instance.
--gatewaybasepath string Gateway base path. (default "/")
--gatewayhostaddr string Gateway host listening address. (default "0.0.0.0:7000")
--grpchostaddr string gRPC host listening address. (default "/ip4/0.0.0.0/tcp/5002")
--grpcwebproxyaddr string gRPC webproxy listening address. (default "0.0.0.0:6002")
--ipfsapiaddr string IPFS API endpoint multiaddress. (Optional, only needed if FFS is used) (default "/ip4/127.0.0.1/tcp/5001")
--lotushost string Lotus client API endpoint multiaddress. (default "/ip4/127.0.0.1/tcp/1234")
--lotusmasteraddr string Existing wallet address in Lotus to be used as source of funding for new FFS instances. (Optional)
--lotustoken string Lotus API authorization token. This flag or --lotustoken file are mandatory.
--lotustokenfile string Path of a file that contains the Lotus API authorization token.
--maxminddbfolder string Path of the folder containing GeoLite2-City.mmdb (default ".")
--mongodb string Mongo database name. (if --mongouri is used, is mandatory
--mongouri string Mongo URI to connect to MongoDB database. (Optional: if empty, will use Badger)
--repopath string Path of the repository where Powergate state will be saved. (default "~/.powergate")
--walletinitialfund int FFS initial funding transaction amount in attoFIL received by --lotusmasteraddr. (if set) (default 250000000000000000)
```
## Localnet mode
Expand Down Expand Up @@ -185,15 +203,26 @@ Terminal 2:
```bash
make build
❯ head -c 700 </dev/urandom > myfile
❯ pow ffs create
> Instance created with id 0ac0fb4d-581c-4276-bd90-a9aa30dd4cb4 and token 883f57b1-4e66-47f8-b291-7cf8b10f6370
❯ pow ffs stage -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 myfile
> Success! Cached file in FFS hot storage with cid: QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm
❯ pow ffs config push -w -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm
> Success! Pushed cid config for QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm to FFS with job id: 966dcb44-9ef4-4d2a-9c90-a8103c77c354
JOB ID STATUS
966dcb44-9ef4-4d2a-9c90-a8103c77c354 Success
❯ pow ffs get -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm myfile2
❯ pow admin profile create
{
"authEntry": {
"id": "c06382e0-2021-4234-be53-6e07a8d40065",
"token": "883f57b1-4e66-47f8-b291-7cf8b10f6370"
}
}
❯ pow data stage -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 myfile
{
"cid": "QmQJxVtp61Y7UrdjUKuWvse3TxGHaPDyA7RobrBhFwqcBM"
}
❯ pow config apply -w -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm
{
"jobId": "b4110048-5367-4ae5-8508-709bf7969748"
}
JOB ID | STATUS | MINER | PRICE | DEAL STATUS
---------------------------------------+--------------------+--------+----------+--------------------
b4110048-5367-4ae5-8508-709bf7969748 | JOB_STATUS_SUCCESS | | |
| | f01000 | 62500000 | StorageDealActive
❯ pow data get -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm myfile2
> Success! Data written to myfile2
```
Expand Down
21 changes: 21 additions & 0 deletions api/client/admin/admin.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
package admin

import (
proto "github.com/textileio/powergate/proto/admin/v1"
)

// Admin provides access to Powergate admin APIs.
type Admin struct {
StorageJobs *StorageJobs
Profiles *Profiles
Wallet *Wallet
}

// NewAdmin creates a new admin API.
func NewAdmin(client proto.PowergateAdminServiceClient) *Admin {
return &Admin{
StorageJobs: &StorageJobs{client: client},
Profiles: &Profiles{client: client},
Wallet: &Wallet{client: client},
}
}
22 changes: 22 additions & 0 deletions api/client/admin/profiles.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package admin

import (
"context"

proto "github.com/textileio/powergate/proto/admin/v1"
)

// Profiles provides access to Powergate admin storage profile APIs.
type Profiles struct {
client proto.PowergateAdminServiceClient
}

// CreateStorageProfile creates a new Powergate storage profile, returning the instance ID and auth token.
func (p *Profiles) CreateStorageProfile(ctx context.Context) (*proto.CreateStorageProfileResponse, error) {
return p.client.CreateStorageProfile(ctx, &proto.CreateStorageProfileRequest{})
}

// StorageProfiles returns a list of existing API instances.
func (p *Profiles) StorageProfiles(ctx context.Context) (*proto.StorageProfilesResponse, error) {
return p.client.StorageProfiles(ctx, &proto.StorageProfilesRequest{})
}
99 changes: 99 additions & 0 deletions api/client/admin/storagejobs.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
package admin

import (
"context"

proto "github.com/textileio/powergate/proto/admin/v1"
)

// StorageJobs provides access to Powergate jobs admin APIs.
type StorageJobs struct {
client proto.PowergateAdminServiceClient
}

type storageJobsConfig struct {
ProfileID string
Cids []string
}

// StorageJobsOption configures a storageJobsConfig.
type StorageJobsOption = func(*storageJobsConfig)

// WithProfileID filters the results to the specified profile.
func WithProfileID(profileID string) StorageJobsOption {
return func(conf *storageJobsConfig) {
conf.ProfileID = profileID
}
}

// WithCids filters the results to the specified data cids.
func WithCids(cids ...string) StorageJobsOption {
return func(conf *storageJobsConfig) {
conf.Cids = cids
}
}

// Queued returns a list of queued storage jobs.
func (j *StorageJobs) Queued(ctx context.Context, opts ...StorageJobsOption) (*proto.QueuedStorageJobsResponse, error) {
conf := &storageJobsConfig{}
for _, opt := range opts {
opt(conf)
}
req := &proto.QueuedStorageJobsRequest{
ProfileId: conf.ProfileID,
Cids: conf.Cids,
}
return j.client.QueuedStorageJobs(ctx, req)
}

// Executing returns a list of executing storage jobs.
func (j *StorageJobs) Executing(ctx context.Context, opts ...StorageJobsOption) (*proto.ExecutingStorageJobsResponse, error) {
conf := &storageJobsConfig{}
for _, opt := range opts {
opt(conf)
}
req := &proto.ExecutingStorageJobsRequest{
ProfileId: conf.ProfileID,
Cids: conf.Cids,
}
return j.client.ExecutingStorageJobs(ctx, req)
}

// LatestFinal returns a list of latest final storage jobs.
func (j *StorageJobs) LatestFinal(ctx context.Context, opts ...StorageJobsOption) (*proto.LatestFinalStorageJobsResponse, error) {
conf := &storageJobsConfig{}
for _, opt := range opts {
opt(conf)
}
req := &proto.LatestFinalStorageJobsRequest{
ProfileId: conf.ProfileID,
Cids: conf.Cids,
}
return j.client.LatestFinalStorageJobs(ctx, req)
}

// LatestSuccessful returns a list of latest successful storage jobs.
func (j *StorageJobs) LatestSuccessful(ctx context.Context, opts ...StorageJobsOption) (*proto.LatestSuccessfulStorageJobsResponse, error) {
conf := &storageJobsConfig{}
for _, opt := range opts {
opt(conf)
}
req := &proto.LatestSuccessfulStorageJobsRequest{
ProfileId: conf.ProfileID,
Cids: conf.Cids,
}
return j.client.LatestSuccessfulStorageJobs(ctx, req)
}

// Summary returns a summary of storage jobs.
func (j *StorageJobs) Summary(ctx context.Context, opts ...StorageJobsOption) (*proto.StorageJobsSummaryResponse, error) {
conf := &storageJobsConfig{}
for _, opt := range opts {
opt(conf)
}
req := &proto.StorageJobsSummaryRequest{
ProfileId: conf.ProfileID,
Cids: conf.Cids,
}
return j.client.StorageJobsSummary(ctx, req)
}
36 changes: 36 additions & 0 deletions api/client/admin/wallet.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
package admin

import (
"context"
"math/big"

proto "github.com/textileio/powergate/proto/admin/v1"
)

// Wallet provides access to Powergate wallet admin APIs.
type Wallet struct {
client proto.PowergateAdminServiceClient
}

// NewAddress creates a new address.
func (w *Wallet) NewAddress(ctx context.Context, addrType string) (*proto.NewAddressResponse, error) {
req := &proto.NewAddressRequest{
AddressType: addrType,
}
return w.client.NewAddress(ctx, req)
}

// Addresses lists all addresses associated with this Powergate.
func (w *Wallet) Addresses(ctx context.Context) (*proto.AddressesResponse, error) {
return w.client.Addresses(ctx, &proto.AddressesRequest{})
}

// SendFil sends FIL from an address associated with this Powergate to any other address.
func (w *Wallet) SendFil(ctx context.Context, from, to string, amount *big.Int) (*proto.SendFilResponse, error) {
req := &proto.SendFilRequest{
From: from,
To: to,
Amount: amount.String(),
}
return w.client.SendFil(ctx, req)
}
Loading

0 comments on commit 363d03b

Please sign in to comment.