Release v00.60.12.100.750.1 2023-03-14

Release package of current v00.60.XX development operations

UPDATED: 2023-03-14

Final Interim Release, pending further testing and expansion

Future development on this script set is moving to GitHub Project R8X_mgmt_cli_API_bash_scripts

v00.60.12

v00.60.12 New Objects Supported

v00.60.12 New JSON exports

• Added support for Data Center objects: Data Center Servers, Data Center Objects, and Data Center Queries for json export
• Added support for custom application-site objects via generic object capture json export
• Added support for custom application-site objects via generic object capture and array evaluation to generate custom application-sites json export

v00.60.12 New JSON and CSV exports

• Added support for Global Properties special object/properties for json and csv export, when exporting domain other than "System Data", or on SMS
• Added support for Policy Settings special object/properties for json and csv export, when exporting domain other than "System Data", or on SMS
• Added support for API Settings special object/properties for json and csv export, when exporting domain "System Data" (also on SMS using domain "System Data")
• Addes support for Radius Server and Radius Group objects for API version 1.9 and later [R81.20 GA], for all operations
• Addes support for Repository Script objects for API version 1.9 and later [R81.20 GA], for all special object operations
• Addes support for SmartTasks objects for API version 1.9 and later [R81.20 GA], for all operations

v00.60.12 New CSV exports

• Added support for application-site objects url-list and additional-categories elements-CSV files (like group members), done in special objects export script or when enabling export of Critical Performance Impacting (CPI) objects
• Added support for custom application-site objects via generic object capture and array evaluation to generate custom application-sites csv export and import, two variants
  • first export variant for CSV import with elements-CSV files (see next New Objects Supported) -- NOT A CPI object!
  • second export variant for CSV import alternative, with up to 20 url-list entries and 10 additional-categories in the exported file
• Added support for custom application-site objects via generic object capture and array evaluation for actual additional-categories and url-list elements-CSV files (like group members), for csv export and import
• Added support for hosts that have NAT configured and that do not have NAT configured to explicit files for easier handling, for csv operations. The original hosts and hosts_NO_NAT operations are still available. Import should either utilize: hosts, hosts_NO_NAT, or hosts_with_NAT and hosts_without_NAT files. JSON export is possible for the object; however, it is disabled to avoid need for more complicated CSV export, as all the data is in the normal hosts file.

v00.60.12 Operational Changes

• Added Command Line Parameters to handle specific domains: "System Data" and "Global", --domain-System-Data|--dSD|--dsd and --domain-Global|--dG|--dg respectively, to handle issues with operational scripts and passing quoted parameters with spaces, as well as easier domain specific execution.
• Added support for export of special objects and properties to json
• Added support for basic plumbing for delete, export, import, set/update, rename, and augment CSV files for special objects and properties via CSV
• Added support for per object | special object/properties specific control of utilization of "details-level", "ignore-errors", "ignore-warnings"
• Modified CSV key value sets exported by default for application-site objects
• Added information more detailed error handling mgmt_cli and JQ calls, to help with identification of problems and performance related limitations
• Added object_operations script files for MDSM with max object limit configuration for 100 objects
• Added CLI parameters to determine handling of Critical Performance Impacting (CPI) objects, [--DO-CPI | --Override-Critical-Performance-Impact] or [--NO-CPI | --NO-Critical-Performance-Impact], like application-site objects with > 10,000 Check Point provided objects to handle. Default mode is to exclude CPI objects from export operations [--NO-CPI | --NO-Critical-Performance-Impact]
• Added script variants to handle special objects for object export, all domain objects export, and all domains objects CSV export.
• Added handler for json extraction of specific objects from a larger set based on export of a reference key value from a generic object query
• Harmonization of the json file slurp operation across different export functions and implementation of some common procedures based on redundant implementations
• Corrections of object association with different main script operations, like export, import, delete, etc.; to ensure that the correct things will work or get skipped
• Added files for operational export of minimum necessary exports for import under the root of the script:
  • _minimum_export.sh
  • _minimum_exports_with_some_do_cpi.sh
  • _minimum_system_data_exports_with_some_do_cpi.sh
• Modified CheckAPIKeepAlive to limit impact of mgmt_cli keep alive calls by checking the last time the procedure was called and only executing an actual mgmt_cli keep alive action if the current default interval of 60 seconds has passed since the last execution; otherwise, make a quick note in logs and continue. A future command line parameter may be added to control the actual interval between required mgmt_cli keep alive executions.
• Reorganized layout of Object Definition Data to make it easier to see variants when reviewing script code
• Cosmetic changes to enhance the flow of operation display, especially when utilization -v (Verbose) logging mode
• Homogenized and harmonized how the routines handling CSV export of complex objects based on generic object queries and associated arrays to the complex object operate, to reduce the number of places to adjust certain processing methods.

OVERVEIW

Operates in the devops.dev folder structure and provides templates with subscripts common to other scripts in the set. Includes example scripts for Check Point Management API based actions for Export, Import, Set [Update], Rename to New-Name, and Delete, and scripts for Zero Lock Session identification and cleanup/removal. Additional scripts examples for MDSM MDM operations (show domains on host) and CSV manipulation tools.

Rough documentation provided in the MD and TSV files (some have Microsoft Excel files analogues).

Identified limitations with certain object types or operations are in the LIMITATIONS_and_CAVEATS.md

To deploy and utilize, it is strongly recommended to copy the "devops.dev.v00.60.12.100.750.tgz" package to /var/log/__customer and untar-gzip (e.g. tar -xvf devops.dev.v00.60.12.100.750.tgz /var/log/__customer/ )

Tested on R81.20 SMS R8X Management API version 1.9.

Running full scripts with the "--help" option will provide command line options to run the script.

It is strongly recommended to think about credentials used for API operations, especially against other management hosts. Establishing a SmartConsole administrator account that utilizes API-KEY as authentication and then setting the api-key for the operation and using that in the script calls with --api-key option.

Execution of scripts and their capabilities will depend on the authority of the SmartConsole administrator user account authenticated, not the local Gaia OS administrator account.

Development, extension, and refinement continue, and this may not be the last v00.60.xx release.

THANKS

Thank you to those who have assisted with feedback and utilization reports and issues.

Additional Documentation

• README
• What's New
• HOW TO Guidance
• Limitations and Caveats
• Supported Objects TSV
• Explanation of the Supported Objects TSV
• File and Folder Purpose and Utilization
• Check Point Software Technologies Management API Reference
• Check Point Software Technologies APIs

Related Projects on GitHub

• Consolidated mgmt_cli bash Script solutions (Under Development)
• Limited Export of mgmt_cli based Policy and Layers export and import scripts
• Windows Scripts for Check Point Operations

CAVEATS

This release package now works with R81.20.

• Tested successfully on R81.20 GA T627 JHF 8
• MDSM testing pending due to infrastructure technical issues with MDSM hosts

(**) Issues with performance throttling and maximum object limits for show operations for large data sets, e.g. application-site objects

With exception of the --MaaS (Smart-1 Cloud) authentication functionality and support for objects lsm-gateways and lsm-clusters, this interim release should work as expected and provide working results for all other objects supported and authentication methods.

--MaaS (Smart-1 Cloud) operation authentication and action should work, and are tested, but still "questionable" due to technical issues with access to Smart-1 Cloud tenant on developer side, so any testing and feedback is greatly appreciated.

lsm-gateways and lsm-clusters require more CSV related refinement, but full export of JSON should work as required, the issue is what to export for CSV inclusion to make working import sets. lsm-gateways has a presumptive CSV export and also an additional CSV export with additional information that will not import (basically raw view of JSON exported), and is identified as DO_NOT_IMPORT moniker on CSV file.

QUICK START

To quickly start working with the scripts, do the following.

1. Create the working __customer folder under /var/log, if that does not exist and configure

   mkdir /var/log/__customer

   chmod 775 /var/log/__customer

   cd /var/log/__customer

2. Download the release tgz file and deploy to a work folder on the target management host, like /var/log/__customer, the folder should be under the /var/log folder to ensure survival during upgrades

3. Expand the TGZ file, e.g.

   Example: tar -xvf devops.dev.{version}.tgz

   tar -xvf devops.dev.v00.60.12.100.750.tgz

4. Goto to the export import folder

   cd ./devops.dev/objects.wip/export_import.wip

5. Execute desired script with help parameter to show command options

   Example: ./cli_api_export_objects_to_csv.sh --help

QUICK START SCRIPT FILES

The following scripts are in the root of the folder ./devops.dev and will execute a minimu export for import on the local managemnet server (SMS). Other scripts for more detailed operations and testing are located under the ./devops.dev/objects.wip/object_operations folder.

Script Files:

• _minimum_exports.sh : Execute the author's bare minimum export operations to enable import for SMS, Critical Performance Impact (CPI) objects, like all application-sites are not exported.
• _minimum_exports_with_some_do_cpi.sh : RECOMMENDED! Execute the author's bare minimum export operations to enable import for SMS, with some commands utilizing the CLI parameter to execute with Critical Performance Impact (CPI) objects.
• _minimum_system_data_exports_with_some_do_cpi.sh : Execute the author's standard export operations using domain "System Data" for SMS, with some commands utilizing the CLI parameter to execute with Critical Performance Impact (CPI) objects.

QUICK START FOR UPDATING

To quickly start working with the scripts if there is an older version installed, do the following.

1. Download the release tgz file or the devops.dev.only.{version}.tgz file and deploy to a work folder on the target management host, like /var/log/__customer, the folder should be under the /var/log folder to ensure survival during upgrades

2. Delete or rename the existing devops.dev folder

   Example: rm /var/log/__customer/devops.dev
   or Example: mv /var/log/__customer/devops.dev /var/log/__customer/devops.dev.old

   rm /var/log/__customer/devops.dev
   or
   mv /var/log/__customer/devops.dev /var/log/__customer/devops.dev.old

3. Expand the TGZ file, e.g.

   Example: tar -xvf devops.dev.only.{version}.tgz
   Or Example: tar -xvf devops.dev.{version}.tgz

   tar -xvf devops.dev.only.v00.60.12.100.750.tgz
   or
   tar -xvf devops.dev.v00.60.12.100.750.tgz

4. Goto to the export import folder

   cd ./objects.wip/export_import.wip

5. Execute desired script with help parameter to show command options

   Example: ./cli_api_export_objects_to_csv.sh --help

TIPS AND TRICKS

Adding this section regarding approach, especailly with respect to performance related limitations that are encountered on Multi-Domain Security Management (MDSM).

HOW TO DETERMINE THE OPERATIONAL --MAXOBJECTS VALUE

Specifically for MDSM it may be necessary to tweak the execution CLI parameter for --MAXOBJECTS X, which for MDSM is set for 250 objects while for SMS is set for 500, the absolute maximum value for "limit" in a mgmt_cli show call. The easiest way to check what is possible on the target MDSM Multi-Domain Server (MDS) host, is executing a few direct mgmt_cli commands looking for the first success value. Starting at a limit value of 250 objects, work down in 100, 50, or 25 increments to find where there is a success output.

Example, start at 250, check 150, then 125, 100 would have been next:

      [Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 250 offset 0 details-level "full" -f json --conn-timeout 600
      {
      "code" : "generic_error",
      "message" : "Error 502. The Management API service is not available. Please check that the Management API server is up and running."
      }

      [Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 150 offset 0 details-level "full" -f json --conn-timeout 600
      {
      "code" : "generic_error",
      "message" : "Error 502. The Management API service is not available. Please check that the Management API server is up and running."
      }

      [Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 125 offset 0 details-level "full" -f json --conn-timeout 600 | tail
            "iso-8601" : "2022-02-25T15:32-0600"
            },
            "creator" : "System"
      },
      "read-only" : true
      } ],
      "from" : 1,
      "to" : 125,
      "total" : 10052
      }
      [Expert@yourhostname:0]#

Based on the above example, adding --OVERRIDEMAXOBJECTS --MAXOBJECTS 125 to the command line execution parameters should ensure proper execution and completion; however, the execution increment will produce ome fun numbers in the files generated. Using --OVERRIDEMAXOBJECTS --MAXOBJECTS 100 may be better, but does require more execution cycles.

Key File Hashes

devops.dev.v00.60.12.100.750.tgz

• MD5 : 77987DF68CB579B5EFF07D2BD3656A7D
• SHA-1 : 4106D1BB334D24CC2F1C9E8181BB9BB7CCA28F24
• SHA-256 : FE94C1F3458D0FB7B72E5DE4B5BCEBCD39E6C8FC4EBB956F23FCE88C590FBB95

Generated by MD5 & SHA Checksum Utility

devops.dev.only.v00.60.12.100.750.tgz

• MD5 : 0F79CC3A4E4AE5B154F7821DAA3D3723
• SHA-1 : 2FCFDAF45A762D4CA71E7671C3E841BC5A169117
• SHA-256 : C5A800F455D183A44AA85FDC47D2AAE24E57B9E43C82E42D6410EE9CDF10A9A4

Generated by MD5 & SHA Checksum Utility