-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#527: Update JSON generator to work with vt's LBDataHolder #528
#527: Update JSON generator to work with vt's LBDataHolder #528
Conversation
…d entity field missing
…ases example compressed data files
…ile_schema against incoming vt new schema
…unications from & to
3d6aae5
to
497ee80
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for changes @tlamonthezie
Requesting now a SNL review @nlslatt @lifflander due to connection with vt |
8e73870
to
c39aecf
Compare
c39aecf
to
ecff71d
Compare
#531 now included in the PR |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The newly generated synthetic-blocks
data works with vt 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM (and works with vt tests)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is there a python script to "copy" json files by reading, decompressing, converting to a dict, converting back to a string, compressing, and writing? This seems very inefficient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello @nlslatt this script was in fact very helpful to rename keys in some compressed datasets.
The script can be called for example to de-compress a dataset, then you can update the dataset using a text editor, then call the script again to compress.
It was particularly helpful in this issue to rename id
to seq_id
in some compressed datasets (in particular tests datasets)
For future schema updates, it might be useful to keep this script
Fixes #527
Fixes #531
This PR provides the following changes;
id
support in entities1.2 The
Object
class -> __index renamed asseq_id
, + addition of apacked_id
fieldcollection_id
field2.1 Object class: Add the
collection_id
field to the attributes of theObject
class, addobjgroup_id
field to unused params.2.2 JSON Generator: Enable the
collection_id
field to be set by the JSON generator (JSONDataFilesMaker) (with interactive mode support)2.3 set collection_id automatically with some default value of 7 if object is added as migratable to a rank (collection_id required by vt for migratable objects so required for the LBAF Json writer).
with the help of the new
TaskSpecification
typed dict defining bothtime
(required) andcollection_id
(not required)entity
node inside the communicationfrom
orto
nodesid
byseq_id
(considering small ids sequences are in factseg_id
's) regenerate to get data in communiaction endpoints nodes and add some generator config fileshared_block_id
byshared_id
everywhereNote: this PR is also needed to adapt to the new LB Data files schema defined by VT PR 2243
IMPORTANT: vt-tv must be updated as well to support
seq_id
as object identifiers.Else we have to generate entity
id
values in LBAF if missing in input data