Skip to content
This repository has been archived by the owner on Sep 30, 2024. It is now read-only.

All Sourcegraph components can export uniform, structured logs #33192

Closed
42 of 54 tasks
taylorsperry opened this issue Mar 29, 2022 · 6 comments
Closed
42 of 54 tasks

All Sourcegraph components can export uniform, structured logs #33192

taylorsperry opened this issue Mar 29, 2022 · 6 comments
Assignees
@bobheadxi
Labels
roadmap Issue tracking a product-eng roadmap item tracking

Comments

@taylorsperry
Copy link
Contributor

taylorsperry commented Mar 29, 2022 •
edited by bobheadxi
Loading

Problem to solve

Today, log15 is used throughout the Sourcegraph codebase for logging. It is mostly sufficient, but has some issues:

  • Strange conventions: log levels are dbug, eror, etc. which is a common source of confusion, and causes interop issues with external tools
  • Unmaintained: most of log15's commit activity is from over 5 years ago, and the repository has a number of abandoned open PRs. We are unlikely to run into log4j-style security risk, but this is still a potential risk.
  • Performance: logging libraries like https://github.com/uber-go/zap are ~1000% to ~5000% faster than log15 (reference) - at Sourcegraph's logging volume this is a likely a non-negligible difference

We also have issues such as extensive use of global logging, which does not include any structured metadata, and extensive use of unstructured logs in general. This leads to a low signal to noise ratio, and a lack of ability to filter logs in a useful manner.

We can take this opportunity to improve tie-in with OpenTracing standards, and improve our sensitive data redaction in logging.

Measure of success

Solution summary

We will propose a complete migration to zap (which would work for Cloud as well as on-prem) and encourage use of passing loggers instances around to enable better structured logging by attaching metadata to logger instances and log entries that can be propagated. Part of this will likely include banning the use of other printing/logging packages other than the chosen logging library.

Artifacts:

What specific customers are we iterating on the problem and solution with?

Internal Sourcegraph developers.

Impact on use cases

This effort contributes to the company-wide effort to improve Observability.

Delivery plan

  • Investigate existing proposals, issues, and problem space details
  • MVP implementation and proposal is prepared details
  • Initial rounds of implementation and migration are handled by DevX
  • Roll out JSON logs in Cloud: https://github.com/sourcegraph/sourcegraph/issues/34599 and ensure the format works (update)
  • Set up tooling to avoid migration regressions and communicate new recommendations to #dev-backend
  • Allocate some migrations to GitStart and ramp them up to perform migrations effectively
  • Support teams in performing their own migrations, and address feedback/documentation issues continuously

Tracked issues

@unassigned

Completed

@bobheadxi: 8.00d

Completed: 8.00d

@burmudar: 6.00d

Completed: 5.00d

@davejrt: 3.00d

Completed: 3.00d

@gitstart-sourcegraph: 8.50d

Completed: 4.50d

@jhchabran

Completed

@marekweb: 1.00d

Completed: 1.00d

@varungandhi-src

Legend

  • 👩 Customer issue
  • 🐛 Bug
  • 🧶 Technical debt
  • 🎩 Quality of life
  • 🛠️ Roadmap
  • 🕵️ Spike
  • 🔒 Security issue
  • 🙆 Stretch goal
@bobheadxi bobheadxi added roadmap Issue tracking a product-eng roadmap item tracking team/devx labels Mar 30, 2022
@bobheadxi bobheadxi changed the title All Sourcegraph components export uniform, structured logs that can be instrumented All Sourcegraph services export uniform, structured logs that can be instrumented Mar 30, 2022
@bobheadxi bobheadxi changed the title All Sourcegraph services export uniform, structured logs that can be instrumented All Sourcegraph components can export uniform, structured logs that can be instrumented Mar 30, 2022
@bobheadxi bobheadxi changed the title All Sourcegraph components can export uniform, structured logs that can be instrumented All Sourcegraph components can export uniform, structured logs Mar 30, 2022
@bobheadxi bobheadxi self-assigned this Mar 30, 2022
@bobheadxi bobheadxi mentioned this issue Mar 30, 2022
Closed
@bobheadxi
Copy link
Member

Brainstorming and feedback discussion: https://github.com/sourcegraph/sourcegraph/discussions/33248

@bobheadxi bobheadxi mentioned this issue Apr 5, 2022
Open
@davejrt davejrt self-assigned this Apr 6, 2022
@taylorsperry taylorsperry self-assigned this Apr 8, 2022
@bobheadxi
Copy link
Member

Migration dashboard: https://k8s.sgdev.org/insights/dashboards/ZGFzaGJvYXJkOnsiSWRUeXBlIjoiY3VzdG9tIiwiQXJnIjo3MjcyNTN9

This was referenced Apr 21, 2022
Closed
Closed
bobheadxi referenced this issue Apr 21, 2022
@bobheadxi @efritz
lib/log: initial implementation (#33956)
dfeb07a 
This introduces a new logging library we plan to enforce as the standard for logging at Sourcegraph - see https://github.com/sourcegraph/sourcegraph/issues/33192 for more details.

Co-authored-by: Eric Fritz <eric@eric-fritz.com>
This was referenced Apr 21, 2022
Merged
Merged
Merged
Closed
@sourcegraph-bot sourcegraph-bot mentioned this issue Apr 27, 2022
Closed
14 tasks
@taylorsperry taylorsperry mentioned this issue May 2, 2022
Closed
1 task
@vrto
Copy link
Contributor

vrto commented May 11, 2022

FYI

I have been looking into OpenTelemetry as we can potentially leverage an existing standard for log data. The summary of my findings is available in this document.

From my findings, I think that our work in the logging improvements seems well-aligned with OpenTelemetry’s log models, and we’ve been mindful about committing to this format early on. I recommend we proceed as planned.

@taylorsperry
Copy link
Contributor Author

Progress: We have a new logging library that encodes some best practices:

  • Safe, strongly-typed ways to add fields to logs
  • Easy-to-use ways to build contextful loggers by attaching metadata to all logs emitted by a logger (e.g. for tracing events and actions taken by subsytems)
  • Restricting the ability to emit context-less logs (no global loggers)

How to add logging and Add observability have been added to the docs.

We are working on migrating logs to this new library. See
Code Insights for Migration progress (approximate).

The next step is to investigate which migrations we can delegate to Gitstart.

@bobheadxi bobheadxi mentioned this issue May 27, 2022
Merged
@bobheadxi
Copy link
Member

bobheadxi commented Jun 8, 2022 •
edited
Loading

Note that the code for this has moved to https://github.com/sourcegraph/log, as of https://github.com/sourcegraph/sourcegraph/pull/36834. There are currently no plans to move the docs, etc and we will probably continue to refer to it as lib/log for some time.

@bobheadxi bobheadxi mentioned this issue Jun 18, 2022
Closed
@sourcegraph-bot sourcegraph-bot mentioned this issue Jun 18, 2022
Closed
4 tasks
@bobheadxi bobheadxi mentioned this issue Jun 20, 2022
Closed
@bobheadxi
Copy link
Member

bobheadxi commented Jun 23, 2022 •
edited
Loading

I am marking this as DONE - the initial goals have been achieved, the API and featureset has stabilized, and the remaining work to bring migration to 100% and address data redaction will be tracked separately in https://github.com/sourcegraph/sourcegraph/issues/37586

@bobheadxi bobheadxi mentioned this issue Jun 23, 2022
Closed
5 tasks
@sourcegraph-bot sourcegraph-bot mentioned this issue Jun 28, 2022
Closed
12 tasks
@bobheadxi bobheadxi mentioned this issue Jun 28, 2022
Closed
Closed
26 tasks
Closed
Closed
17 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@bobheadxi bobheadxi

Labels
roadmap Issue tracking a product-eng roadmap item tracking
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@davejrt @vrto @bobheadxi @taylorsperry