Proposal: Enhanced User Logging System on Akash Console

[HoomanDgtl]

Proposal: Log Forwarding Integration for the Akash Network: Empowering User-Controlled Logging Solutions

Introduction

This proposal represents a strategic pivot in our approach to implementing logging solutions for the Akash Network. Following in-depth discussions with the Akash core team, we have refined our strategy to better align with the network's principles of decentralization and user needs while maintaining simplicity and effectiveness.

Initially, we proposed developing a self-hosted logging infrastructure using Grafana, Prometheus, and Loki. However, discussions with the core team highlighted the following key insights:

• Many users already have established relationships with various logging providers.
• Concerns were raised about overloading provider services for streaming logs.
• Leveraging existing battle-tested solutions provides immediate value.
• Multiple Akash customers prefer SaaS logging solutions like Datadog and Grafana Cloud.

Based on these insights, we revised our approach to focus on integrating robust log forwarding capabilities, allowing users to utilize their preferred logging solutions while maintaining full control over their data and costs.

---

Problem Statement

The Akash Network console currently lacks a persistent logging system, leading to significant challenges:

• Limited Log Availability: Only live logs are accessible; there is no historical retention, making debugging difficult after a system crash or unexpected behavior.
• Ineffective Debugging: Without log history, identifying the root cause of issues is challenging, especially for complex deployments.
• Community Support Barriers: New users or developers lack the ability to share logs for troubleshooting within the community, complicating the onboarding process.

---

Proposed Solution

We propose integrating comprehensive log forwarding capabilities into the Akash console to:

• Enable persistent log storage through users' preferred logging solutions.
• Provide a simple, configuration-based setup process.
• Allow users to maintain control over their logging costs and retention policies.
• Leverage existing, battle-tested logging infrastructure.

---

Implementation Details

Console Integration

Settings Page Enhancement:

• Add a new "Logging Configuration" section to the console settings.
• Allow users to input credentials for supported logging destinations.
• Validate credentials before saving.
• Securely store logging configuration.

SDL Builder Integration:

• Add an "Enable Logging" checkbox in the SDL builder section.
• Automatically inject the logging sidecar configuration into the deployment when enabled.
• Allow users to modify logging parameters through the UI.

---

Technical Implementation

Logging Sidecar Service:

• Automatically add a logging sidecar container to deployments when logging is enabled.
• Configure the sidecar to collect logs from all containers in the deployment.
• Forward logs to the specified logging destination using stored credentials.
• Implement retry mechanisms and buffering for reliable log delivery.

Supported Logging Destinations (Initial Phase):

• Datadog
• Grafana Cloud
• Elastic Logstash
  (Expand support to additional platforms based on user feedback.)

---

Implementation Timeline (2-3 Months)

Phase 1: Frontend Development

• Design and implement console UI components.
• Add a logging configuration section in the settings page.
• Develop credential management and validation workflows.
• Create a logging toggle and basic configuration options in the SDL builder.

Phase 2: Backend Development

• Design and implement secure credential storage.
• Develop the logging sidecar container configuration.
• Create a system for injecting logging configurations into deployments.
• Implement initial logging destinations.

Phase 3: Testing and Documentation

• Conduct end-to-end testing of logging integration.
• Create user documentation and troubleshooting guides.
• Develop documentation for adding support for additional logging destinations.

---

Expected Outcomes

This revised logging integration approach will provide:

• Professional-grade logging capabilities through users' preferred solutions.
• Simplified deployment monitoring and troubleshooting.
• Cost-effective solutions leveraging existing infrastructure.
• Improved developer experience and debugging capabilities.
• Enhanced support for production workloads.

This solution aligns with Akash Network's principles of decentralization and user empowerment while providing immediate value to the community.

---

Budget Estimate

The following budget reflects the comprehensive scope of work required:

Category	Hours	Description
Research & MVP Testing	150	User needs identification and MVP conceptualization.
UI/UX Design	80	Designing user-friendly configuration flows.
Frontend Development	150	UI components and SDL builder enhancements.
Base Container Development	180	Building a robust logging sidecar container.
Data-Logging Integration	120	Integration with Datadog, Grafana Cloud, and Elastic Logstash.
Production Testing	100	Thorough testing, documentation, and refinements.

Total Hours: 780
Hourly Rate: $100
Total Budget: $78k

This budget ensures a professional-grade implementation of log forwarding integration, addressing user pain points while aligning with Akash Network's values.

Governance Timeline

• Discussion & Feedback: ongoing
• On-Chain Voting: Targeting Thursday/Friday this week

Link to POC Video: click here
Previous Prop & plan (just for record): click here

---

Thank you for considering this proposal. We welcome your feedback and community support on our updated plan. If you have any questions, please feel free to ask below.

Thank you

[anilmurty]

Hey @HoomanDgtl (Piyush and Chai),

Thanks very much for initiating this and for coming up with the POC and presenting it to the core team. To echo what you said above, not being able to capture, retain and query past logs easily is a very acute painpoint for our customers and solving it is one of 3-4 top level initiatives on the Console Roadmap (alongside CC payments, Alerting/ Notifications and better/ easier workflows for non-container native developers).

To recap our discussion (between the core team and your team):
While this solution is great - there are a couple challenges with storing logs on Akash providers:

1. Data stored on Akash providers is only persisted for the life of the lease today (because Akash is primarily a compute network/ cloud and not a storage network/ cloud). So if the lease that is used to store the logs is closed the user logs are lost which defeats the purpose of logging.
2. Multiple customers of Akash have directly indicated that they prefer using SaaS logging solutions like Datadog and Grafana Cloud. This is likely driven by the fact that many of these customers use traditional (centralized) cloud services before considering or moving to akash and they would not want to disrupt all their workflows (and monitoring solution is a key part of those workflows). So it is essential that the first tenant logging solution Akash implements support "log forwarding" to those common SaaS logging services, so as to ease the transition for customers coming to Akash from traditional clouds.
3. While you can technically store the logs AND ALSO forward them to a SaaS logging service, that would be unnecessary extra cost for the user/ customer.
4. There are concerns from the core team about overloading provider services for streaming logs. The last thing we want to do is cause provider services to have issues, because it would impact compute workloads, bid engine etc.

With those things in mind, the recommendation from the core team is to:

1. Modify this proposal to be a "Log Forwarding" service that forwards logs to common/ leading SaaS logging solutions that are used by users who use centralized clouds today (like AWS, Azure, GCP).
2. Identify 2-3 such SaaS logging services (my recommendation is: Datadog, Grafana cloud and Elastic Logstash, in that order of priority) and integrate them to start with and then add more over time (based on customer/ user requests)
3. The integration itself could be one where the user is able to enter auth credentials for their user account in their preferred logging service one time in Console, save it and have it be used for any/ all leases where they wish to forward logs. This can very likely be in the Settings page where the user will select (from a growing list of supported external logging services) a logging service and then enter the required auth info (API Key, Secret etc). Then when the user creates a deployment they will check a box to say "enable logging" - which can be either at deployment creation time or after the deployment is done. In either case, we would need to let the user be able to enable/ disable logging even after the lease is created and ideally would want to let the user be able to do this per lease and not across the board for all leases.

I'll note that the above proposed solution does require that the user be familiar with external logging solutions, how to set them up and how to use them to query logs so this would not be for less technical users (like those who use vercel) - but for now I think this is a good solution to at least address the logging problem for users who understand infrastructure (which I would argue is a vast majority of akash users today). Over time we can look at offering a more seamless experience where the user does not need to configure anything and just pays for a service in Console to "take care of logging" (whether that involves storing logs on akash providers or some other decentralized storage provider like StorJ or Filecoin or to a centralized logging service.

Thanks again for working on this critical problem and we look forward to the updated "Log Forwarding" proposal from you

[HoomanDgtl]

Thank you, @anilmurty and team, for your constructive feedback and suggestions. We will follow up with an updated proposal within this week.

Best regards,
Piyush
Hooman Digital

[pcfreak30]

I am also very interested in this. if this can go to endpoints for granfana's stack self hosted, that would be awesome as well. I would take advantage of this as soon as I could. I am actively R&D'ing some rather complex infrastructure and this will be needed long term.

[HoomanDgtl]

Thank you, @pcfreak30, for taking the time to review our proposal. We will make sure that your request is considered in the final product we’re building.

As Anil mentioned, our initial focus will be on supporting log forwarding to Datadog, Grafana Cloud, and Elastic Logstash, with plans to expand support to additional platforms based on customer and user feedback over time.

[HoomanDgtl]

The proposal is scheduled to go live onchain tomorrow.
thank you