skywire-cli log collecting command

[mrpalide]

Did you run make format && make check? Yes

Fixes #_

Changes:

• add log-collecting script

How to test this PR:

• run a visor on test env as root for create node-info.json and node-info.sha
• wait 5 minute to update uptime tracker
• collecting logs and node-info by skywire-cli log collecting -d <count-of-days-for-getting-logs> -e <environment>

Example:

./skywire-cli log collecting -d 5 -e test

This command get last 5 days log files and node-info files from test env. Default value for -e (env) is prod and for -d (duration) is 1.

[0pcom]

Do we need to use an interval for getting transport logs, or is yesterday is sufficient?

what is the default log rotation interval? We shouldn't expect very old logs to be there, beyond the default interval.

so we could by default fetch yesterday's, and have a flag to be able to get days before that, up to the default log rotation interval

How we should check or use transport logs?

we pull the logs from the server and client, or the visor on both ends of the transport. We compare these to make sure they agree about the data sent and received.

In the future we may do a 1-byte hash of the actual data sent and received

But so long as the numbers match on both ends of the transport, everything is good

if the numbers don't match, then this is indicative of some issue we would need to look into. And its unlikely to be a singular occurrence.

If the logs can't be fetched from the client side, which is altogether likely considering skywire or the machine running skywire (i.e. someone's desktop / vpn consumer) may not run continuously; then we can just ignore that or take it as an honest record. Is there no other way to tell that a transport did exist, from the service end?

we need to just confirm that if possible rather than allow someone to make false transport logs

to be improved upon later

What is reward logic?

A list of the public keys, ranked by bandwidth handled according to the transport logs is
divided as evenly as possible into three parts. If the list is not divisible by 3 with no remainder, then the middle tier can contain the extra keys.

so if there are 10 visors; 3 in the top tier, 3 in the bottom, and 4 in the middle.

if there are 11, then the middle tier can contain 5 keys

If a visor processed no bandwidth, we can put its public key in the bottom tier.

And if some visors have equal bandwidth, according to the transport logs, we should simply randomize the order of those keys which had the same bandwidth. Then divide the list as evenly as possible.

If you have objection to this, simply state your alternative idea. I think its better to populate the middle tier rather than overflowing to the top and bottom ones.

I've merged your PR here:

skycoin/dmsg#198

Please note that the condition for the node-info.json (system survey) to be generated, which may be on my autoconfig PR, is subject to the presence of the reward address being set in the local path as reward.txt

additionally

• Never reward any node that does not have the system survey. (It won't be possible because we won't have the reward address)
• Do not reward the genesis address, which is used as a default for faster testing of reward file setting
• We need eventually a reward notification system via skychat. This will require a cli interface for skychat; or otherwise a way to send many notifications automatically and simultaneously

[0pcom]

Testing this, it runs but isn't collecting very much. It fetched the logs and survey from my machine

I know why it isn't collecting very many logs. We are querying visors which are not of a version we should expect to have the dmsghttp log server running.

Can you check the versions of the visors before attempting to collect the logs?

This also does not seem to implement the ping. That is ok, since we can't ping them yesterday but we are collecting the logs from yesterday. We may not integrate it right here..

If we simply get a list of the keys to be rewarded we could script ping after the fact.

I don't see much in terms of log reconciliation either. But since my visor was the only one which logs were collected for, I would not expect that to be done.

I recommend further tests after the release.