Gigamon (ECOINT-9)

[Mrudula-Oruganti-Gigamon]

Integration with Gigamon

What does this PR do?

Integration with Gigamon

Motivation

What inspired you to submit this pull request?

Review checklist

• PR has a meaningful title or PR has the no-changelog label attached
• Feature or bugfix has tests
• Git history is clean
• If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo
• If this PR includes a log pipeline, please add a description describing the remappers and processors.

Additional Notes

Anything else we should know when reviewing?

[Mrudula-Oruganti-Gigamon]

@vishalshah-dd , i have made respective changes as you suggested. Could you please review and let us know ?

[vishalshah-dd]

Requesting changes:

• The namespacing of Gigamon on the custom facets was done on the name and not the path.
• The grok expression to extract the date isn't working in the pipeline. It should be on the ts attribute. I don't think this was tested in your custom pipeline, can you please make the changes and test before requesting a review for that?

[vishalshah-dd]

This isn't right, it should be ts, the source attribute where we're applying the grok expression. The message today has the entire contents. The sample should also be changed to be just the value: Wed Jul 31 12:05:39 2024.

Once done, you should also add a timestamp remapper with the source attribute as whatever you extract out of the grok parser.

Can you make these changes and test it with your test pipeline? To ensure that the dates you're sending in the logs gets correctly set within Datadog? We drop old logs so please make sure the timestamp you're testing with is within the same day you send it for ingestion.

[vishalshah-dd]

May have gotten missed in the back-and-forth, but I mentioned it's the path that needs namespacing with gigamon. prefix and not the name. The paths need to be unique across all integrations, and that's why we ask contributors to namespace the custom facets.

This one should look like:

  - facetType: list
    groups:
      - gigamon
    name: App ID
    path: gigamon.app_id
    source: log
    type: string

With changing the facet, you'll also need a remapper to map app_id to gigamon.app_id:

    - type: attribute-remapper
      name: Map 'app_id' to 'gigamon.app_id'
      enabled: true
      sources:
        - app_id
      sourceType: attribute
      target: gigamon.app_id
      targetType: attribute
      preserveSource: false
      overrideOnConflict: false

These remappers would be needed for each custom facet that we add the gigamon. prefix to

[Mrudula-Oruganti-Gigamon]

@vishalshah-dd , Thanks for the details. Apologize for delay. We were occupied in customer deliverables. Will work on these and will get back to you. Before that, I do have couple of questions on your two queries:

1. Is it mandatory to use timestamp ? can we skip that one?
2. Should we manually change the facet values there? As per your suggestion, we should change the path with "gigamon." prefix in the facets. However, in the facet creation, i am unable to edit the path as it is coming as drop down. Am i missing any ? Please help.
3. Rempapper, i am able to make the changes. however, first we need to change the facet right. Any help on this ?

[Mrudula-Oruganti-Gigamon]

@vishalshah-dd , should we do the below facet type maunally in the yaml file ?

• facetType: list
  groups:
  • gigamon
    name: App ID
    path: gigamon.app_id
    source: log
    type: string

[emarsha94]

Hey @Mrudula-Oruganti-Gigamon jumping in here

1. Timestamp is mandatory. Please treat any requested changes from logs as mandatory moving forward.
2. You will need to delete each facet. Next, you will remap all the attributes added as facets under the group Gigamon. (see 3) From there, after the remappers have been added, you will create new facets using the remapped attributes.
3. You will first remap each attribute then you will create the facets on the newly remapped attributes (attribute paths). ex remapped attribute: gigamon.app_id then new facet will be created for gigamon.app_id

[Mrudula-Oruganti-Gigamon]

Thanks @emarsha94 , Will work on it and will update it.

[Mrudula-Oruganti-Gigamon]

@emarsha94 , could you please review #2356 this thread and confirm ? Thanks.

[Mrudula-Oruganti-Gigamon]

Thank you team for the clarification on facets. We tested it with the sample ID "gigamon.app_id," and we're now able to see the facet as shown below. We've also written the "grok parser" for the timestamp and remapped it with "gigamon.ts." The logs are now displaying correctly, as shown in the attached screenshot.

Could you please confirm if this is the expected behaviour? Attaching the screenshots for reference.

We'll be adding a few more facets and will submit the PR as soon as possible. Thank you for all your help.

[Mrudula-Oruganti-Gigamon]

[emarsha94]

Hi @Mrudula-Oruganti-Gigamon please use this doc which walks through how to remap ts to be the official timestamp: https://docs.datadoghq.com/developers/integrations/log_pipeline/#log-pipeline-requirements

This doc walks through exactly how the standard attributes like timestamp should be remapped

[Mrudula-Oruganti-Gigamon]

Hi @vishalshah-dd / @emarsha94 , i have made respective changes. could you please review and let us know if any changes are required ?

[vishalshah-dd]

• You've changed all existing facets to be under "Gigamon" and the gigamon. path prefix. This was only supposed to be done to the facets you originally had under "Gigamon".
• I see a bunch of new facets added that weren't there before, some of these should not be under "Gigamon" group and are standard facets (one example: gigamon.src_port this should be mapped to network.client.port. You can see a list of standard facets for logs here. Any facet that you have that has the same behavior as these standard facets should reuse the same paths mentioned in the linked doc, and should not be under the "Gigamon" facet group
• Need to remove the timestamp facet, and keep the value as just ts and then have a date remapper on ts. This should not be saved as gigamon.ts since it's a standard attribute in Datadog

[vishalshah-dd]

Wait sorry, why did you change these to be under the "Gigamon" group and namespace? The only changes to add the gigamon. prefix to the path were the facets that were under the "Gigamon" group. These standard facets you had before were supposed to be unchanged.

I found this version of the file before you made the changes. You had all the other facets correctly handled under the respective facet groups like Web Access, and DNS. The only facets that needed appending a gigamon. prefix to the path were app_name, dns_host, and dns_name. All the other facets were meant to remain unchanged

[Mrudula-Oruganti-Gigamon]

Thanks @vishalshah-dd for the clarification. I will do respective changes. I will keep the standard attributes as such and will add the gigamon. prefix for the custom attributes.

[emarsha94]

Hi @Mrudula-Oruganti-Gigamon could you please revert your last commit? Once that is done, please let me know and I will advise on next steps.

As Vishal said, it looks like there were a lot of changes that weren't required or requested that have been added to the yaml files. Please revert and I will re-comment on what needs to be changed.

[Mrudula-Oruganti-Gigamon]

@emarsha94 , I have made changes as per Vishal's suggestion in the latest commit and have added few more attributes and facets as per his statements.
If i revert back those commit, i may need to perform all the other changes like timestamp, adding new facets again. Do you still want me to revert to the one which i did 3months back?
Is there any issue with the latest build ? Please help.

[emarsha94]

Hi Mrudula, it looks like there were a lot of changes that were not asked and additional facets added that don't belong in the groups they were added to. At this point, I believe the next best step is to revert the most recent commit then follow Vishals comments very carefully: #2356 (review)

-You've changed all existing facets to be under "Gigamon" and the gigamon. path prefix. This was only supposed to be done to the facets you originally had under "Gigamon".

• I see a bunch of new facets added that weren't there before, some of these should not be under "Gigamon" group and are standard facets (one example: gigamon.src_port this should be mapped to network.client.port. You can see a list of standard facets for logs here. Any facet that you have that has the same behavior as these standard facets should reuse the same paths mentioned in the linked doc, and should not be under the "Gigamon" facet group
• Need to remove the timestamp facet, and keep the value as just ts and then have a date remapper on ts. This should not be saved as gigamon.ts since it's a standard attribute in Datadog

Please read the above and review the linked documents and update as the above instructs. Examples were provided in this comment on exactly how the formatting should look.

[Mrudula-Oruganti-Gigamon]

gigamon.yaml.txt

@emarsha94 , That's true. This is what we have done in our latest commit which we did 4days ago. Do we need to do "submit for review" everytime we commit from github? Attaching the updated yaml file (in .txt format) for reference.

[Mrudula-Oruganti-Gigamon]

@emarsha94 , the comments that were given by @vishalshah-dd and you have been addressed . Please refer to the comments at the below link :

#2356 (comment)

This should be part of the build:
58b166c

please let me know if there is anything else.

[vishalshah-dd]

This one too, we don't need to have a facet, and the timestamp isn't supposed to be under a Gigamon prefix or facet group since it's a Datadog standard attribute

[Mrudula-Oruganti-Gigamon]

Sure, Please confirm the below steps that i did for timestamp.

1. Created "Date ReMapper" as shown in the screenshot
2. Created "grok parser" as shown in the screenshot.

[Mrudula-Oruganti-Gigamon]

Hi @vishalshah-dd , i have made the changes as below. Please correct if my understanding is wrong:

1. Mapped Standard Attributes: 11 - I have mapped these attributes with our "Gigamon Attributes" but have not added them in "gigamon" facet group and have not added "gigamon" prefix. I have used "Remapper" and one "user agent parser" to map these in which the "attribute" name is "gigamon actual attribute" and the "target attribute" is "datadog standard attribute".

2. Mapped Timestamp with Date Remapper and have written Grok parser to parse the value. Attached the screenshots in the above Comment .

3. Mapped Gigamon (Custom) Attributes: 13 - I have mapped these attributes with "gigamon" prefix and have added them to "gigamon" facet group. I have used "remapper" to map these in which the "attribute" name is " gigamon actual attribute" and the "target attribute" is with "gigamon prefix along with gigamon actual attribute".

Please let me know if any changes are required. Thanks.

[Mrudula-Oruganti-Gigamon]

@vishalshah-dd / @emarsha94 , Could you please review and let us know whether we are good with the latest submission?

[Mrudula-Oruganti-Gigamon]

Updated as per comments

[Mrudula-Oruganti-Gigamon]

@emarsha94 / @vishalshah-dd , Could you please let us know the status ?

[emarsha94]

Hi @Mrudula-Oruganti-Gigamon I am at a team event until Friday this week. I will review the PR Friday or Monday

[Mrudula-Oruganti-Gigamon]

Hi @Mrudula-Oruganti-Gigamon I am at a team event until Friday this week. I will review the PR Friday or Monday

Sure @emarsha94 . Thanks for the update.