As a Node Operator, I want to specify an alternate file paths for 1 or more archive products

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

Node Operator, Data User

💪 Motivation

...so that I can load data into the registry where we compressed the archive product and removed the uncompressed version from the filesystem prior to harvesting the metadata

📖 Additional Details

No response

Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

Proposed design:

1. New “--compressed-mapping” argument to Harvest - CSV table that includes LIDVID,compressed_filename

2. Harvest will attempt to load all products in bundle/directory like normal

3. If product LIDVID == LIDVID from CSV table
   a. Add additional step to read the compressed_filename and add the following metadata to the Registry:

            "ops:Data_File_Compressed_Info/ops:creation_date_time": { "type": "date" },
            "ops:Data_File_Compressed_Info/ops:file_ref": { "type": "keyword" },
            "ops:Data_File_Compressed_Info/ops:file_name": { "type": "keyword" },
            "ops:Data_File_Compressed_Info/ops:file_size": { "type": "long" },
            "ops:Data_File_Compressed_Info/ops:md5_checksum": { "type": "keyword" },
            "ops:Data_File_Compressed_Info/ops:mime_type": { "type": "keyword" },
   

   b. If //File/file_name not found, decompress compressed_filename into temp space, get the applicable ops:Data_File_Info metadata, and cleanup

   • Note: This should support .zip, .tar.gz, .gz, and .fz

Some questions:

• Should this be separate ops:Data_File_Compressed_Info class or under ops:Data_File_Info as something like ops:Data_File_Info/ops:compressed_file_ref?