add JSON generator script test

[DanhLuong]

No description provided.

[tunglxfast]

If weight tests aren't finished, you should remove them. Using @ignore will make misunderstanding that those tests are done.

[ake2l]

looks good

[ake2l]

@DanhLuong when you are actively working on, you should at least checkin and push once a day

[DanhLuong]

Yes, I'll keep tracking and update everyday for this task when working on it.

[DanhLuong]

After the weight check test, it will have make report in console:
REPORT
Please check the input and output to verify fuction of data.wgt.csv file

Test 1: gender.wgt.csv
Input:
gender[Gender=Male, percentage=0.5]
gender[Gender=Female, percentage=0.3]
gender[Gender=Other, percentage=0.2]
Output:
Total male in user list: 523, percentage: 0.523
Total female in user list: 288, percentage: 0.288
Total other in user list: 189, percentage: 0.189

Test 2: skill.wgt.csv
Input:
skill[Name=Java, count=12, percentage=12/88=0.136]
skill[Name=C#, count=10, percentage=10/88=0.114]
skill[Name=Javascript, count=36, percentage=36/88=0.409]
skill[Name=Python, count=30, percentage=130/88=0.341]
Output:
Total java in skill list: 480, percentage: 0.13977868375072802
Total c# in skill list: 398, percentage: 0.11589982527664532
Total js in skill list: 1401, percentage: 0.4079790331974374
Total python in skill list: 1155, percentage: 0.3363424577751893

[DanhLuong]

I come to test the case for using data from other DB: H2, Postgres, Oraccle, DB2, MySQL to import to JSON writen to mongoDB. It is not complete now. First try to set-up some to local container, still fail when connecting to H2 DB.

Up to now, the 3 test commited ba77b12 is Integration Test with Github Action Mongo DB Service only (passed quickcheck.yaml), not in E2ET. Now, I join #413 with Anh Tung so I will decrease time for doing this task daily.

[ake2l]

first it needs to work locally and a h2 db is a in memory db only exists during runtime when used like this , in case you want to read data from you need to create schema first and put data in as shown in many of our demos ... you can also use postgres .... apart from this , this is needs to be done as well ...

• Data Iteration

  • Configure the iteration process to iterate over a collection filled with data from a previous Benerator run.
  • Test different consumers, including LoggingExporter, MongoDB,JSON, Preview, SQL, CSV, mongodb.updater(), and mongodb.inserter('test'), to ensure proper data interaction with different collections.
  • Utilize selectors with different MongoDB queries to fetch specific data subsets during iteration. This needs to be a bit more extensive , we should also test negative scenarios to check how its working with the benerator, like query data from different collection as specified in type , crosscollection queries to check data relation in different collection . We should use selectors in iterate node , variable and attribute as well ... to see how the behavior is and also check how it is working with subselector

• Negative Cases

  • Design tests to handle scenarios where the collection being iterated is empty.
  • Implement tests to handle cases where the targeted collection does not exist.

• Separate Benerator Runs

  • Execute the data generation process in one Benerator run, ensuring the generated data is stored in the Mongo DB collection.
  • Conduct a separate Benerator run specifically for data iteration, validating the correctness of the iteration process independently from the generated data.

• Validation and Reporting

  • Validate that the iteration process successfully retrieves and interacts with the desired data subsets.
  • Ensure that the tests for negative cases handle empty collections and non-existing targeted collections gracefully.
  • Document any issues, bugs, or unexpected behavior encountered during the testing process.
  • Generate a report summarizing the results of the tests and any identified areas for improvement.

[DanhLuong]

Yes, I try to setup (connect to DB and create schema, add some data type with generator referred from demos) and make it a source to generate JSON write to MongoDB consumer. I will use all source DB type: H2, Postgres, Oraccle, DB2, MySQL for this test (in local container first, then integrate into GitHub Action Service)

[DanhLuong]

Only 1 topic left: Utilize selectors with different MongoDB queries to fetch specific data subsets during iteration, please advise me more for this, we don't have demo before. Also have a look at these test and report. Any comment for improve please let me know

[DanhLuong]

Complete tests as checklists. 2 Test on JSON Iteration and Query MongoDB (please check comments on ben.xml file for details) is on #[Add-more-tests-for-MongoDB]. For MongoDB Querry tests, need more tests for cases for: aggregate, count, mapReduce. For current tests on this branch, need some adjusts to pass ci, then it's OK

[DanhLuong]

quickcheck.yaml all passed. @tunglxfast and @dangsg please review. Any improvement, please discuss with me. Thanks