Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[log] optimize console log format #3499

Merged
merged 20 commits into from
Aug 31, 2022
Merged

Conversation

huof6829
Copy link
Contributor

@huof6829 huof6829 commented Jul 5, 2022

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #3476

Type of change

Please delete options that are not relevant.

  • [] Bug fix (non-breaking change which fixes an issue)
  • [] New feature (non-breaking change which adds functionality)
  • [] Code refactor or improvement
  • [] Breaking change (fix or feature that would cause a new or changed behavior of existing functionality)
  • [] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • [] make test
  • [] fullsync
  • [] Other test (please specify)

Test Configuration:

  • Firmware version:
  • Hardware:
  • Toolchain:
  • SDK:

Checklist:

  • [] My code follows the style guidelines of this project
  • [] I have performed a self-review of my code
  • [] I have commented my code, particularly in hard-to-understand areas
  • [] I have made corresponding changes to the documentation
  • [] My changes generate no new warnings
  • [] I have added tests that prove my fix is effective or that my feature works
  • [] New and existing unit tests pass locally with my changes
  • [] Any dependent changes have been merged and published in downstream modules

@huof6829 huof6829 requested a review from a team as a code owner July 5, 2022 11:39
@huof6829
Copy link
Contributor Author

huof6829 commented Jul 5, 2022

Just modify the starting log format for viewing. Leave the zap log because most codes use zap function. After zerolog merged, we can replace zap with zerolog.

output in cmdline:
stdout1

/var/log/s.log is json formatter:
slog1

@huof6829
Copy link
Contributor Author

huof6829 commented Jul 5, 2022

One note: "github.com/iotexproject/go-p2p" use zap.Logger now. If change to zerolog, this project should also change together.

p2p.SetLogger(log.L())

@Liuhaai
Copy link
Member

Liuhaai commented Jul 5, 2022

Could this format of logs be indexed?

@dustinxie
Copy link
Member

see comment in #3476, we need to achieve:

  1. the output to log file is still same json format, for ELK log collection/analysis purpose
  2. on top of that, if it's possible to make the output to commandline window better

replace with zerolog does not seem a good solution to achieve both

@millken
Copy link
Contributor

millken commented Jul 7, 2022

zap Encoder supportted customize log format,you can implement an encoder to replace default console encoder.

@codecov
Copy link

codecov bot commented Jul 7, 2022

Codecov Report

Merging #3499 (ce2faee) into master (a20e489) will decrease coverage by 1.07%.
The diff coverage is 70.03%.

@@            Coverage Diff             @@
##           master    #3499      +/-   ##
==========================================
- Coverage   75.43%   74.35%   -1.08%     
==========================================
  Files         247      259      +12     
  Lines       22845    23536     +691     
==========================================
+ Hits        17233    17501     +268     
- Misses       4685     5107     +422     
- Partials      927      928       +1     
Impacted Files Coverage Δ
action/action_deserializer.go 57.14% <ø> (ø)
action/protocol/poll/nativestaking.go 41.08% <0.00%> (-0.65%) ⬇️
action/protocol/poll/staking_command.go 10.71% <0.00%> (ø)
action/protocol/staking/read_state.go 15.38% <0.00%> (ø)
action/protocol/vote/probationlist.go 87.50% <ø> (ø)
api/blocklistener.go 70.73% <0.00%> (ø)
api/websocket.go 5.17% <0.00%> (-0.19%) ⬇️
blockchain/block/block_deserializer.go 71.15% <ø> (ø)
blockchain/blockchain.go 0.89% <0.00%> (ø)
blockchain/filedao/filedao_legacy.go 85.80% <ø> (ø)
... and 105 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@huof6829
Copy link
Contributor Author

huof6829 commented Jul 7, 2022

see comment in #3476, we need to achieve:

  1. the output to log file is still same json format, for ELK log collection/analysis purpose
  2. on top of that, if it's possible to make the output to commandline window better

replace with zerolog does not seem a good solution to achieve both

the log file is json format. colored printing in cmmandline. see the picture above

@huof6829
Copy link
Contributor Author

huof6829 commented Jul 8, 2022

see comment in #3476, we need to achieve:

  1. the output to log file is still same json format, for ELK log collection/analysis purpose
  2. on top of that, if it's possible to make the output to commandline window better

replace with zerolog does not seem a good solution to achieve both

Because of ELK, the log file format can't change. I'll try to find the humanly-printed setting by zap console encoder.

@huof6829
Copy link
Contributor Author

huof6829 commented Jul 14, 2022

add zap console encoder
zap-console

s.log keeps the same.

@huof6829 huof6829 changed the title [log] zerolog for better log format [log] optimize console log format Aug 16, 2022
@raullenchai
Copy link
Member

ping @dustinxie @CoderZhi @Liuhaai - pls take another look

pkg/log/log.go Outdated
var buildOpts []zap.Option
buildOpts = append(buildOpts, coreOpt)
buildOpts = append(buildOpts, opts...)
logger, err := cfg.Zap.Build(buildOpts...)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cfg.Zap.Build(coreOpt, opts)?

Copy link
Contributor Author

@huof6829 huof6829 Aug 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no. build error because type of opt is []zap.Option,

pkg/log/log.go Outdated
zapcore.NewConsoleEncoder(consoleCfg.EncoderConfig),
zapcore.Lock(os.Stderr),
zap.InfoLevel))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so looks like add a new console encoder to process the output in human-friendly format?

i assume you already run locally to verify the log format in the file has remained the same?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, add console encoder. I run in test-net, log format is the same with older

@sonarcloud
Copy link

sonarcloud bot commented Aug 31, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@dustinxie dustinxie merged commit 672320a into iotexproject:master Aug 31, 2022
@raullenchai
Copy link
Member

Thanks for making this one @huof6829 @dustinxie

@huof6829 huof6829 deleted the zerolog branch September 7, 2022 00:59
dustinxie pushed a commit that referenced this pull request Oct 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

A better log format
5 participants