Add support for setting custom fields in LogEntryEventBuilder class

[pankaj0509]

Hi Team,

I want to use managed packaged version of nebula logger. I want to know is there any plugin available
so that i can create my custom method in LogEntryEventBuilder class.

I am able to so that in unmanaged package and my expected functionality is working fine .

but i want to archive the same in managed package version of nebula so i need plugin information to explore the possibilities.

Pankaj

[jongpie]

Hi @pankaj0509 - that is not feasible for the managed package, nor recommended for the unlocked package as it will make it difficult to upgrade to new releases. Can you share some info on what the new method does that you're adding?

[codejester90]

Hey @jongpie let me provide some additional context about what we are working on with @pankaj0509.
We have a requirement to log some additional data in the Log Entry. This data is rather context- and organization-specific, so it is not a good candidate for a Feature Request. Not having these fields in the log will unfortunately be a deal breaker for us, so if we want to use Nebula, we have to find some way to include them.

I was thinking about a generic way of adding additional fields without impacting the core logic of the Nebula Framework. That's why Pankaj was asking for new builder methods for the LogEntryEventBuilder and new fields on the Log Entry object.

We found some workarounds that allow us to use the unmanaged package, but even in that case, we need to make some minor changes to Nebula to add new fields to the Log Entry. As this does not modify the existing metadata, it will allow us to keep up with the latest releases of Nebula without any problems. Let me describe my solution:

• I want to extend the LogMessage class with some builder methods so that any additional field is added to a Message__c field as a JSON structure. Let's assume that I want to add MyAdditionalField__c to the LogEntry. Then the message would look like:

"My standard error message ## { "MyAdditionalField__c": "My Custom Value For Additional Field" }"

Then, in the before insert plugin, I search for the JSON structure in the message and map the fields to my custom fields added to the log message. After that, I remove all data from the error message (starting from ## or any other specific set of characters).

It might work fine, but as you can see, this is not the cleanest solution ever created, and furthermore, it requires creating custom fields, which makes it impossible in managed package distribution.

So after this long introduction, let me ask a question:
Do you think that it would make sense to enable having custom fields on the log entry? Do you see any chance of having it in Nebula in the foreseeable future? Maybe there is any existing feature in the current solution that I am missing here, which would be a good option for this use case?

I was thinking about two potential, generic solutions for that. Here are my thoughts:

1. Enable users to use any custom object to store additional data for the Log Entry. Users would have to provide information about the name of the custom object and the fields that could contain log data in some configuration. In the LogEntryEventBuilder, there could be a new method

setCustomField(fieldName, value)

Custom data could be saved in JSON (some new field on LogEntryEvent__e) and then in after insert plugin dynamically mapped to custom fields. Log Entry should also have a new field - Id of the custom record containing additional log data. The problem here would be presenting that data - they will be not available on standard page layouts, so they would require some component that dynamically retrieves the additional log data and presents them.

2. Adding a few fields for user purposes like CustomField1__c, CustomField2__c, CustomField3__c. This solution also has significant drawbacks. It should allow users to present the data in a more meaningful way, so again, it would be necessary to create a custom component to present them, as on standard components the standard label would be meaningless. It would for sure require some setting containing a mapping between field API name, and field meaningful name, so that we can create a builder method like in point 1.

[jongpie]

Hi @codejester90, thanks for the explanation! The only existing features that might work for your purposes are (in addition to what you've already tried):

• Tagging your log entries - you can use methods like Logger.info('hello, world').addTag('some tag value'); to use Nebula Logger's tagging system, which stores data in the custom objects LoggerTag__c and LogEntryTag__c.
  • Benefits: you can dynamically add 1 or more tag to each log entry record, so you could store multiple values/data points
  • Downsides: this functionality would only store a string value, and it uses additional data storage in your org
• Scenario-based logging - you can use methods like Logger.setScenario(String) and Logger.endScenario(String) to identify & control segments of your code base
  • Downsides: I don't think this really fits your use case, this functionality would only store a string value, and it uses additional data storage in your org

So, tagging is maybe the only existing functionality that would possibly work for your use case, but I think that it would have a lot of downsides in terms of reporting on the data (compared to using custom fields with distinct data types, etc.). Conceptually, I think it does make sense to provide a way to support custom fields in Nebula Logger, there are a lot of projects/orgs that could benefit from having a place to store their own data, in a structured way.

I see this working a little bit differently from the 2 options you provided - I think it should support custom fields on Nebula Logger's own objects (instead of using a separate custom object as described in your 1st option). I built a prototype of this functionality a while back, but never released it - this approach would have 3 changes in Nebula Logger's codebase:

1. A new instance method in LogEntryEventBuilder - setCustomField(field, value) like you suggested (or some similar name, maybe just setField(field, value)). This would be used to set custom fields on LogEntryEvent__e.
2. A new custom metadata type, LoggerFieldMapping__mdt. This would be used to define mappings between LogEntryEvent__e fields and fields on Nebula Logger's custom objects. For example:
   • Your custom field LogEntryEvent__e.SomeField__c maps to Log__c.SomeField__c
   • Your custom field LogEntryEvent__e.AnotherField__c maps to LogEntry__c.AnotherField__c
   • (repeat for each custom field you make)
3. Internal updates in Nebula Logger to read the new CMDT LoggerFieldMapping__mdt and automatically set the mapped fields when creating data in its custom objects (Log__c, LogEntry__c, etc.).

To use the approach, you would then make a few changes in your org

1. Add a custom field on LogEntryEvent__e.

2. Add a corresponding custom field on whichever custom object you want to store the data (Log__c, LogEntry__c, etc.).

3. Add a record in LoggerFieldMapping__mdt for the field mapping so Nebula Logger knows how to map the data.

4. Add something like this in your code to provide the field values:

   Logger.info('hello, world')
       .setField(LogEntryEvent__e.SomeCustomStringField__c, 'my custom field value')
       .setField(LogEntryEvent__e.SomeCustomDateField__c, System.now());

This approach would have a couple of benefits:

• It doesn't increase the data storage used (since it's just adding new fields to existing objects/records, instead of creating new records in a separate custom object)
• You could add any custom field data type
• You could add as many custom fields as you want, with some platform limitations to keep in mind:
  • The platform would possibly run into some issues, if you added a large number of new fields - for example, the Apex CPU time & heap size would increase, which increases the chance of exceeding the transaction limits.
  • There are also platform limits to how many custom fields you can add to a platform event object or custom object
• You would have the flexibility to specify which of Nebula Logger's objects should store the data for each field (via the LoggerFieldMapping__mdt CMDT)
• You would be able to deploy (and store in version control) your LoggerFieldMapping__mdt CMDT records

Let me know what you and @pankaj0509 think of this approach!

[codejester90]

@jongpie, thanks for the quick and detailed response! Let me express my thoughts:

Tagging:
This feature is interesting; it covers one of our requirements, and we are going to use it. However, as you mentioned, it has some limitations.

Your Idea of Implementing Custom Fields:
I like your approach a lot. Initially, I had reservations about adding fields to objects in managed or unmanaged packages, but upon consideration, this solution seems to have more advantages than disadvantages (if any). Therefore, I would consider it a much better idea than having a separate object for this purpose. For instance, it alleviates concerns about data storage usage and displaying values in standard page layouts.

The only aspect that I'm not entirely happy with is the need to create a field both on LogEntryEvent__e and Log__c/LogEntry__c. If there is a mapping in LoggerFieldMapping__mdt, then we could simply have one field on LogEntryEvent__e and store all custom fields as JSON. I understand the necessity of not doing this on Log__c/LogEntry__c due to the benefits derived from existing fields rather than a JSON structure. However, LogEntryEvent__e is only a means to transport the data, and I believe users should not manipulate or maintain this object.

This is an internal object, and thus far, you've protected the API values of it by employing the builder pattern, which is commendable. It theoretically allows you to replace LogEntryEvent__e with something else without impacting the client code. Hence, I suggest maintaining this level of protection. Exposing the internals to the client via:

Logger.info('hello, world')
    .setField(LogEntryEvent__e.SomeCustomStringField__c, 'my custom field value')
    .setField(LogEntryEvent__e.SomeCustomDateField__c, System.now());

I believe that a dynamic JSON LogEntryEvent__e and reference to Log__c/LogEntry__c instead of LogEntryEvent__e could potentially be a good solution.

Summary:
Overall, I really like your idea. It's better than having a separate object, and my only concerns are regarding exposing LogEntryEvent__e. Nevertheless, having this solution in any shape would be a huge benefit for our organization, as it would allow us to replace our legacy solution with your excellent Nebula library.

Please let me know if there's any way we can assist you in making this happen. We would be more than happy to contribute, whether it's through coding, testing, or any other means. Additionally, if there's a timeline for when this could happen, even having it on the beta branch would be great so that we could start testing it in our environments.

[jongpie]

Hi @codejester90, thanks for the feedback! I definitely see your point about LogEntryEvent__e, but 2 considerations that make me nervous about using a single JSON long textarea field on LogEntryEvent__e:

1. It won't scale well in some situations. For example, if someone wanted to have a long textarea field on Log__c and another long textarea field on LogEntry__c (or 2 long textarea fields on Log__c, etc.), then a single long textarea field on LogEntryEvent__e may not be able to store the full value of the JSON data.
2. Some orgs do heavily rely on/integrate with LogEntryEvent__e - I know of a couple of orgs that only use LogEntryEvent__e, they don't store any data in Log__c and LogEntry__c. Instead, they have external systems subscribe to LogEntryEvent__e events - so having an approach that would allow these orgs/projects to add their own custom fields (with specific data types) would help to provide structured data to external systems that have been integrated with Nebula Logger.

I think I'm still leaning towards the approach I outlined of using custom fields on LogEntryEvent__e + custom fields on Log__c/LogEntry__c, but I'm going to try to work on this enhancement more this week, and hopefully finalize an approach. I'll keep you updated on my progress.

If you're interested in testing beta version of the package, that would also be really helpful. The prototype I made a few years ago already has some of the functionality built, so I think I can make fairly quick progress on this enhancement. I might be able to have a beta version ready to test in a week or two.

[codejester90]

@jongpie thanks for the response, I do appreciate the time you dedicated to consider the points I raised

I completely understand your concerns regarding using a single JSON long text area field on LogEntryEvent__e, your explanation makes perfect sense, especially regarding potential scalability issues and the importance of accommodating orgs that heavily rely on/ integrate with LogEntryEvent__e. It's important to keep the solution flexible and robust enough to meet various use cases and integration scenarios. Your approach seems more versatile

I'm super glad to hear that you'll be working on this enhancement further this week, and I'm looking forward to seeing the finalized approach. Your dedication to improving the Nebula Logger is really impressive.

Regarding testing the beta version of the package, count us in! We'd love to participate and provide feedback. Please keep us updated on your progress and let us know when the beta version is available for testing (no pressure though :) )
Thanks again for your collaboration and dedication to developing Nebula Logger :)

[jongpie]

@codejester90 awesome, thanks for the feedback (again!) - I'm moving forward with the approach of having matching fields on LogEntryEvent__e + the target custom object (Log__c, LogEntry__c, or LoggerScenario__c).

I've already made progress on this enhancement, and I'm aiming to have a beta version ready in the next few days. There are 2-3 other smaller releases that I'm going to finish first, then I'll create a PR with this enhancement. I'll let you know when it's ready & available to test out!

[pankaj0509]

@jongpie Awesome to know that Thanks for your update.

[jongpie]

@pankaj0509 and @codejester90 - I have an initial beta of the unlocked package ready to try out! 🥳

• The beta package version ID is 04t5Y000001MkHHQA0. You can install using one of these methods:
  • Use the CLI: sf package install --wait 20 --security-type AdminsOnly --package 04t5Y000001MkHHQA0
  • Use the UI: https://test.salesforce.com/packaging/installPackage.apexp?p0=04t5Y000001MkHHQA0
• I've created PR Added support for custom field mappings in Apex #664 - it contains details on how to setup & use the new functionality

There are a couple of things I may change in the PR, so there may be some breaking changes before this is officially released, but you should be able to use this beta to test out the new functionality in a scratch org or sandbox.

If/when you have a chance to test it out, please let me know if you have any feedback!

[pankaj0509]

@jongpie I implemented the same using Beta version however It is not working . I created one abc__c on LogEntryEvent__e and same custom field i.e abc__c on logEntry__c and i also created one entry to provided CMDT .

It is not working ,Not only this but logs itself are not generating which means no log__c record are generating

as part of our investigation i found one behavior if i unchecked IsEnabled__c on CMDT entry then log __c records will be generating but again our custom field population feature is not working.

is there any other thing that I am missing ? kindly suggest

[jongpie]

Hi @pankaj0509 - could you tell me what field type(s) you're using on LogEntryEvent__e and Log__c?

[pankaj0509]

hi @jongpie thanks for your reply !!. I am using text field type in both LogEntryEvent__e and logEntry__c . For our scenario , We need to populate data in logEntry__c object's custom field.

In essence , let me summarize it for you.

To populate custom field on logEntry__c, We need to use create custom metadata entry on LoggerFieldMapping__mdt but when i create an entry by following to steps mentioned in above with IsEnabled__c is true. log__c records are not generating . However I can see logs in debug logs but no log__c records are generating .

but incase there is not entry in LoggerFieldMapping__mdt or if there is entry in LoggerFieldMapping__mdt with IsEnabled__c is false then log__C records will be created but this new feature won't work.

fyi i am using below link to install nebula.

https://test.salesforce.com/packaging/installPackage.apexp?p0=04t5Y000001MkHHQA0

Your prompt response will be much appreciated . Thanks again for your valuable support .

[jongpie]

Hi @pankaj0509 thanks for the feedback! I was able to reproduce and fix several issues that could occur with the field mappings. Could you install & test this new version?

https://test.salesforce.com/packaging/installPackage.apexp?p0=04t5Y0000027L0EQAU

Hopefully this resolves the issues, but please let me know if you run into any more problems.

[pankaj0509]

@jongpie Thanks, I am on it . Let me give u feedback

[pankaj0509]

@jongpie I have installed nebula using below provided link and it is working as expected. Thanks a ton!!

https://test.salesforce.com/packaging/installPackage.apexp?p0=04t5Y0000027L0EQAU

just out of curiosity , when are you planning to merge new feature changes in main ?

[jongpie]

@pankaj0509 thanks for trying it out & glad to hear it's working as expected! There are still a couple things I want to improve & finalize before this is released, so my hope is to merge this sometime in the coming weeks/next month or so, after I've addressed a few things:

• I've found a few more small bugs that I need to fix - there's still an issue or two related to setting String fields with a value that exceed the custom field's max length, as well as a few other small bugfixes & optimizations that I'd like to make
• Currently, PR Added support for custom field mappings in Apex #664 only includes functionality for setting custom fields in Apex - long-term, I want to also have support for setting custom fields in LWC and Flow. I will probably implement those later in separate PRs, but before merging Added support for custom field mappings in Apex #664, I'd like to first spend some additional planning/prototyping of how I'll make this functionality work in LWC & Flow, just to make sure it will align with what's been built for Apex

In the meantime, if you see any other bugs or have any other feedback, please let me know!

[jongpie]

@pankaj0509 just wanted to (finally) give you an update on this enhancement

• I am currently focused on PR Summer '24 Release (API v61.0) - v4.14.0 #700 to release v4.14.0 - this will not include the custom field mapping changes
• After v4.14.0 is released, I'll finish PR Added support for custom field mappings in Apex #664 to officially release the new Apex method setField()
• Later, I'll add similar functionality to the logger LWC to support setting custom fields in JavaScript (I've built a prototype for this, so I know more or less how to make it work)
• Much later 😅 , I'll try to add the same functionality for Flows, but I think that will be a more complex endeavor

So, I think the setField() functionality you've been testing will finally be available in the coming weeks - I don't have an ETA on when I'll complete the LWC & Flow side, but the Apex side will be ready soon.

[jongpie]

Related discussion in #681.

[jongpie]

@pankaj0509 and @codejester90, another update on this - contrary to my previous comment, I am going to release this today for the unlocked package, and it will then be included in the next managed package release (v4.14.0 - PR #700), which I hope to have released next week.

There are a few small changes to this functionality (compared to the beta version that you've been using). I think the only notable change that could impact your code is that the name of the method setFields(Map<SObjectField, Object>) has been slightly changed - it is now setField (no trailing s). The other method overload, setField(SObjectField, Object), has not changed.

[jongpie]

This has now been released in v4.13.14 of the unlocked package.

[JaskiratSingh29]

We are working on the custom field mapping from Lightning Web Components, we are currently encountering a blocker. The logger component passes details to the controller class, which then creates a new platform event record, adds the details, and executes the save log process. Unfortunately, there is no room for custom logic to somehow retrieve the platform event or the Log Entry record and map our custom fields.

I am open to any suggestions or guidance on how to tackle this blocker. Can you also clear up some of my concerns for logger

1. What is the timeline for having this feature with LWC and Flow?
2. What can be an alternative approach we can have to add the custom field mapping from flow and LWC

[jongpie]

@JaskiratSingh29 in order to make this work in lightning components, there are changes needed in the logger LWC and the Apex class ComponentLogger, where the platform events are created & saved.

What is the timeline for having this feature with LWC and Flow?

There is not a specific timeline yet for adding it to lightning components & Flow (please keep in mind that this is a free project that I maintain as a hobby), but I've created issues #718 and #719 to track the progress of these enhancements...

• I already have a working prototype for adding this functionality for lightning components, but there is still more testing & cleanup needed before it's ready to be released
• I have not worked on adding this to Flow, I anticipate that adding Flow support will be much more difficult than Apex or lightning components

[JaskiratSingh29]

@jongpie thanks for the quick response and the detailed update! I appreciate the effort you’re putting into enhancing the logger functionality for Lightning components and Flow. I completely understand the need for thorough testing and cleanup, and I’m excited about the benefits this feature will bring once it’s ready. I look forward to the progress on issues #718 and #719.

Can you share the early prototype version for Logger with this functionality implemented for LWC?