Skip to content

Release v00.60.12.000.050 2022-10-27 INTERIM
Compare
Choose a tag to compare
@mybasementcloud mybasementcloud released this 28 Oct 01:38
· 5 commits to master since this release
00.60.12.000.050
a55b41b

Release package of current v00.60.XX development operations

UPDATED: 2022-10-27

Interim update

What's New v00.60.12.000

  • Added support for export of special objects and properties to json
  • Added support for basic plumbing for delete, export, import, set/update, rename, and augment CSV files for special objects and properties via CSV
  • Added support for per object | special object/properties specific control of utilization of "details-level", "ignore-errors", "ignore-warnings"
  • Added support for Global Properties special object/properties for json export, when exporting domain other than "System Data", or on SMS
  • Added support for Policy Settings special object/properties for json export, when exporting domain other than "System Data", or on SMS
  • Added support for API Settings special object/properties for json export, when exporting domain "System Data"
  • Modified CSV key value sets exported by default for application-site objects
  • Added support for application-site objects url-list and additional-categories sub-CSV files (like group members), done in special objects export script
  • Added information more detailed error handling mgmt_cli and JQ calls, to help with identification of problems and performance related limitations
  • Added object_operations script files for MDSM with max object limit configuration for 100 objects

OVERVEIW

Operates in the devops.dev folder structure and provides templates with subscripts common to other scripts in the set. Includes example scripts for Check Point Management API based actions for Export, Import, Set [Update], Rename to New-Name, and Delete, and scripts for Zero Lock Session identification and cleanup/removal. Additional scripts examples for MDSM MDM operations (show domains on host) and CSV manipulation tools.

Rough documentation provided in the MD and TSV files (some have Microsoft Excel files analogues).

Identified limitations with certain object types or operations are in the LIMITATIONS_and_CAVEATS.md

To deploy and utilize, it is strongly recommended to copy the "devops.dev.v00.60.12.000.tgz" package to /var/log/__customer and untar-gzip (e.g. tar -xvf devops.dev.v00.60.12.000.tgz /var/log/__customer/ )

Tested on R81.10 MDSM MDS and SMS R8X Management API version 1.8.1.
Tested on R81.20 SMS R8X Management API version 1.9.

Running full scripts with the "--help" option will provide command line options to run the script.

It is strongly recommended to think about credentials used for API operations, especially against other management hosts. Establishing a SmartConsole administrator account that utilizes API-KEY as authentication and then setting the api-key for the operation and using that in the script calls with --api-key option.

Execution of scripts and their capabilities will depend on the authority of the SmartConsole administrator user account authenticated, not the local Gaia OS administrator account.

Development, extension, and refinement continue, and this may not be the last v00.60.xx release.

THANKS

Thank you to those who have assisted with feedback and utilization reports and issues.

CAVEATS

This release package now works with R81.20 EA, but additional testing and work needed, as well as later takes for testing.

  • Tested successfully on R81.10 JHF OT 79 last
  • Tested with partial success on MDSM R81.10 JHF OT 79 (**)
  • Tested successfully on R81.20 EA T570 Public EA

(**) Issues with performance throttling and maximum object limits for show operations for large data sets, e.g. application-site objects

With exception of the --MaaS (Smart-1 Cloud) authentication functionality and support for objects lsm-gateways and lsm-clusters, this interim release should work as expected and provide working results for all other objects supported and authentication methods.

--MaaS (Smart-1 Cloud) operation authentication and action should work, and are tested, but still "questionable" due to technical issues with access to Smart-1 Cloud tenant on developer side, so any testing and feedback is greatly appreciated.

lsm-gateways and lsm-clusters require more CSV related refinement, but full export of JSON should work as required, the issue is what to export for CSV inclusion to make working import sets. lsm-gateways has a presumptive CSV export and also an additional CSV export with additional information that will not import (basically raw view of JSON exported), and is identified as DO_NOT_IMPORT moniker on CSV file.

QUICK START

To quickly start working with the scripts, do the following.

  1. Create the working __customer folder under /var/log, if that does not exist and configure

    mkdir /var/log/__customer

chmod 775 /var/log/__customer

cd /var/log/__customer

  2. Download the release tgz file and deploy to a work folder on the target management host, like /var/log/__customer, the folder should be under the /var/log folder to ensure survival during upgrades

  3. Expand the TGZ file, e.g.

    Example: tar -xvf devops.dev.{version}.tgz

    tar -xvf devops.dev.v00.60.12.000.tgz

  4. Goto to the export import folder

    cd ./objects.wip/export_import.wip

  5. Execute desired script with help parameter to show command options

    Example: ./cli_api_export_objects_to_csv.sh --help

QUICK START FOR UPDATING

To quickly start working with the scripts if there is an older version installed, do the following.

  1. Download the release tgz file or the devops.dev.only.{version}.tgz file and deploy to a work folder on the target management host, like /var/log/__customer, the folder should be under the /var/log folder to ensure survival during upgrades

  2. Delete or rename the existing devops.dev folder

    Example: rm /var/log/__customer/devops.dev
    or Example: mv /var/log/__customer/devops.dev /var/log/__customer/devops.dev.old

    rm /var/log/__customer/devops.dev
    or
    mv /var/log/__customer/devops.dev /var/log/__customer/devops.dev.old

  3. Expand the TGZ file, e.g.

    Example: tar -xvf devops.dev.only.{version}.tgz
    Or Example: tar -xvf devops.dev.{version}.tgz

    tar -xvf devops.dev.only.v00.60.12.000.tgz
    or
    tar -xvf devops.dev.v00.60.12.000.tgz

  4. Goto to the export import folder

    cd ./objects.wip/export_import.wip

  5. Execute desired script with help parameter to show command options

    Example: ./cli_api_export_objects_to_csv.sh --help

Key File Hashes

devops.dev.v00.60.12.000.tgz

  • MD5 : 27C42E32093961B08FA49F5963FCCB1C
  • SHA-1 : BB9897643FA56D62ECF5797B07E4F1C43E9E51E3
  • SHA-256 : 7989F4E20F3D35BEEE530A19F5E40CFF1BE9C0A5810B84B2DB15FA791C6D2CFA

devops.dev.only.v00.60.12.000.tgz

  • MD5 : 519D26CEE84B8591104688A2021543D8
  • SHA-1 : B1C171ACAABC1CFC8B38C6BE27DE5F8631A7E516
  • SHA-256 : 86745CA5F3CDBBBC0A75509D48B31B54F8740F7714AE90B4F1F0081B54D874B7

Generated by MD5 & SHA Checksum Utility @ http://raylin.wordpress.com/downloads/md5-sha-1-checksum-utility

TIPS AND TRICKS

Adding this section regarding approach, especailly with respect to performance related limitations that are encountered on Multi-Domain Security Management (MDSM).

HOW TO DETERMINE THE OPERATIONAL --MAXOBJECTS VALUE

Specifically for MDSM it may be necessary to tweak the execution CLI parameter for --MAXOBJECTS X, which for MDSM is set for 250 objects while for SMS is set for 500, the absolute maximum value for "limit" in a mgmt_cli show call. The easiest way to check what is possible on the target MDSM Multi-Domain Server (MDS) host, is executing a few direct mgmt_cli commands looking for the first success value. Starting at a limit value of 250 objects, work down in 100, 50, or 25 increments to find where there is a success output.

Example, start at 250, check 150, then 125, 100 would have been next:

      [Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 250 offset 0 details-level "full" -f json --conn-timeout 600
      {
      "code" : "generic_error",
      "message" : "Error 502. The Management API service is not available. Please check that the Management API server is up and running."
      }

      [Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 150 offset 0 details-level "full" -f json --conn-timeout 600
      {
      "code" : "generic_error",
      "message" : "Error 502. The Management API service is not available. Please check that the Management API server is up and running."
      }

      [Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 125 offset 0 details-level "full" -f json --conn-timeout 600 | tail
            "iso-8601" : "2022-02-25T15:32-0600"
            },
            "creator" : "System"
      },
      "read-only" : true
      } ],
      "from" : 1,
      "to" : 125,
      "total" : 10052
      }
      [Expert@yourhostname:0]#

Based on the above example, adding --OVERRIDEMAXOBJECTS --MAXOBJECTS 125 to the command line execution parameters should ensure proper execution and completion; however, the execution increment will produce ome fun numbers in the files generated. Using --OVERRIDEMAXOBJECTS --MAXOBJECTS 100 may be better, but does require more execution cycles.

Assets 6
Loading