Reports showing duplicate entries on test success

[BenStreet365]

Apologies in advance if this has already been reported, I thought I saw a similar issue but can't seem to find it.

I have implemented Maester through GitHub Actions using the documentation. The majority of our tests are showing duplicate responses in the results, in both markdown and HTML. Clicking into the failed result shows it was actually a success. This is version 0.2.0 of the Action but I also tested the latest preview.

I can look at the script too if someone with deeper knowledge of the tool is able to point out where the failure could be :)

[BenStreet365]

I found issue #71 reported this issue fixed but I still have the problem, happy to help out where I can.

[BenStreet365]

Trying to provide more information where I can, but I've looked at the logic in one of the simpler checks (EIDSCA.AF01) which seems to all be correct.

I'm not sure if the problem here comes down to the 'Add-MtTestResultDetail' script, as in our environment every result is duplicated. Once for success, once for failure. If the test is something we have failed, a failure is reported twice.

[fdicarlo]

It is happening to me as well for all the tests, it started to happen once I have changed the main.yml file in the pipeline (and unfortunately I was never been able to restore the single occurrence of tests)

[BenStreet365]

I have found more info on this, as below.

This looks to show for most, if not all, EIDSCA checks though I'm not sure why this is happening within the module.

[merill]

Can you please try running again? I just published a new version of the module that should have this cmdlet.

[BenStreet365]

Still seems a problem unfortunately. Here's a sample output we have from the JSON and a visual from the step summary. Whilst now reporting success, the results are still duplicated. All I can notice from the output is how the ScriptBlock line is slightly different.

{
      "Name": "EIDSCA.AF01: Authentication Method - FIDO2 security key - State.",
      "HelpUrl": "https://maester.dev/docs/tests/EIDSCA.AF01",
      "Tag": [
        "EIDSCA",
        "Security",
        "All",
        "EIDSCA.AF01"
      ],
      "Result": "Passed",
      "ScriptBlock": "\r\n        <#\r\n            Check if \"https://graph.microsoft.com/beta/policies/authenticationMethodsPolicy/authenticationMethodConfigurations('Fido2')\"\r\n            .state = 'enabled'\r\n        #>\r\n        Test-MtEidscaControl -CheckId AF01 | Should -Be 'enabled'\r\n    ",
      "ScriptBlockFile": "/home/runner/work/maester/maester/private-tests/tests/EIDSCA/Test-EIDSCA.Generated.Tests.ps1",
      "ErrorRecord": [],
      "Block": "Authentication Method - FIDO2 security key",
      "ResultDetail": {
        "SkippedReason": null,
        "TestResult": "\r\nWell done. The configuration in your tenant and recommended value is *'enabled'* for *policies/authenticationMethodsPolicy/authenticationMethodConfigurations('Fido2')*\r\n",
        "TestSkipped": "",
        "TestDescription": "Whether the FIDO2 security keys is enabled in the tenant.\r\n\r\nenabled\r\n\r\n#### Test script\r\n⁠  \r\nhttps://graph.microsoft.com/beta/policies/authenticationMethodsPolicy/authenticationMethodConfigurations('Fido2')\r\n.state = 'enabled'\r\n  ⁠\r\n\r\n#### Related links\r\n\r\n- [Open in Graph Explorer](https://developer.microsoft.com/en-us/graph/graph-explorer?request=policies/authenticationMethodsPolicy/authenticationMethodConfigurations('Fido2')&method=GET&version=beta&GraphUrl=https://graph.microsoft.com)\r\n- [fido2AuthenticationMethodConfiguration resource type - Microsoft Graph v1.0 | Microsoft Learn](https://learn.microsoft.com/en-us/graph/api/resources/fido2authenticationmethodconfiguration)\r\n\r\n\r\n"
      }
    },
    {
      "Name": "EIDSCA.AF01: Authentication Method - FIDO2 security key - State.",
      "HelpUrl": "https://maester.dev/docs/tests/EIDSCA.AF01",
      "Tag": [
        "EIDSCA",
        "Security",
        "All",
        "EIDSCA.AF01"
      ],
      "Result": "Passed",
      "ScriptBlock": "\n        <#\n            Check if \"https://graph.microsoft.com/beta/policies/authenticationMethodsPolicy/authenticationMethodConfigurations('Fido2')\"\n            .state = 'enabled'\n        #>\n        Test-MtEidscaControl -CheckId AF01 | Should -Be 'enabled'\n    ",
      "ScriptBlockFile": "/home/runner/work/maester/maester/public-tests/tests/EIDSCA/Test-EIDSCA.Generated.Tests.ps1",
      "ErrorRecord": [],
      "Block": "Authentication Method - FIDO2 security key",
      "ResultDetail": {
        "SkippedReason": null,
        "TestResult": "\r\nWell done. The configuration in your tenant and recommended value is *'enabled'* for *policies/authenticationMethodsPolicy/authenticationMethodConfigurations('Fido2')*\r\n",
        "TestSkipped": "",
        "TestDescription": "Whether the FIDO2 security keys is enabled in the tenant.\r\n\r\nenabled\r\n\r\n#### Test script\r\n⁠  \r\nhttps://graph.microsoft.com/beta/policies/authenticationMethodsPolicy/authenticationMethodConfigurations('Fido2')\r\n.state = 'enabled'\r\n  ⁠\r\n\r\n#### Related links\r\n\r\n- [Open in Graph Explorer](https://developer.microsoft.com/en-us/graph/graph-explorer?request=policies/authenticationMethodsPolicy/authenticationMethodConfigurations('Fido2')&method=GET&version=beta&GraphUrl=https://graph.microsoft.com)\r\n- [fido2AuthenticationMethodConfiguration resource type - Microsoft Graph v1.0 | Microsoft Learn](https://learn.microsoft.com/en-us/graph/api/resources/fido2authenticationmethodconfiguration)\r\n\r\n\r\n"
      }
    }

[Oppedijk]

I have the same issue, maybe you can change the "motd" to only MAESTER, and include the version (including build version) as normal text.
Currently it returns the big banner with v0.2 (which probably is the v0.3 version)

[Oppedijk]

I think this bug is related to the import of the Github action, with duplicate tests.

So I run my own tests, and it's combined with all the tests from the main maester project as well.
For instance when I delete one of my own tests, the count of tests only goes down by one.

I think when I delete all my tests, it will still run all the maester supplied ones.

[BenStreet365]

Can confirm this is the case. The default run for the GitHub action includes a checkout for all public tests. Setting include_public_tests to false fixes that!

[Oppedijk]

@BenStreet365 shouldn't this be documented then somewhere?

[BenStreet365]

@Oppedijk Yes, thinking about it that would be a good idea. I'll look at how I can tweak the documentation tomorrow for the GitHub Actions setup.