Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slices optimization #263

Merged
merged 2 commits into from
Jan 19, 2022
Merged

Slices optimization #263

merged 2 commits into from
Jan 19, 2022

Conversation

MykhailoNester
Copy link
Contributor

  • add eTag to all api calls;
  • add new API for slices;
  • add base logic for 412 error handling;
  • add logic for comparing remote chunks slices with local;

@MykhailoNester
- add eTag to all api calls;
a176c80 
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;
@MykhailoNester MykhailoNester requested a review from a team as a code owner January 17, 2022 18:04
@MykhailoNester MykhailoNester merged commit faec027 into feature/certificate-revocation Jan 19, 2022
@MykhailoNester MykhailoNester deleted the feature/slice-optimization branch January 19, 2022 11:09
oleksandrsarapulovgl pushed a commit that referenced this pull request Feb 2, 2022
@MykhailoNester @oleksandrsarapulovgl
Slices optimization (#263)
d38d728 
* - add eTag to all api calls;
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;

* - fix unit test;
MykhailoNester added a commit that referenced this pull request Feb 4, 2022
@MykhailoNester @oleksandrsarapulovgl @craftbyte
Api integration (#276)
08b05ab 
* - add revocation module; (#249)

* - add revocation module;

* - update dependency;

* - setup base network logic with use cases; (#250)

* DCC revocation local storage (#253)

* - add interceptor with mock API service responses; (#254)

* Fix parsing of issuer country (#251)

Closes #245

* - add interceptor with mock API service responses;

Co-authored-by: Anže Jenšterle <anze@anze.dev>

* - add etag header; (#255)

- move loading to worker;

* Implemented locate storage for dcc revocation entries (#256)

* - generate 3 types of dcc hashes; (#259)

* Feature/revocation local storage (#258)

* - update network API responses and models; (#257)

- add new usecase;

* Feature/revocation hash lookup (#260)

* Feature/connect api with db (#261)

* Connect API with DB. Update downloading flow;

- add logic for kid loading and storing into DB;
- add API mapper;
- fix base url;
- store last modified in pref.;

* Connect API with DB. Update downloading flow;

- package structure update;
- add new partition structure;
- db update;

* Connect API with DB. Update downloading flow;

- update API model according to new spec;
- update DB columns;
- refactor downloading logic;

* - fix tests;

* - todo item;
- fix test;

* Slices optimization (#263)

* - add eTag to all api calls;
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;

* - fix unit test;

* Slices part; (#265)

* Slices part;

- save slices to DB;
- delete expired data from DB;

* - fix test;

* DB lookup; (#266)

* DB lookup;

- add usecase for hash lookup, mode check and loading data from db;

* - fix tests;

* - add z value to db;
- add enum converter for db;
- update responses with test data;

* Feature/certificate revocation local layer filtering update (#267)

* import bloom filter (#268)

* - import bloom filter with unit tests;
- fix datetime format in DB;
- update KID to base64Url;

* - add bloom filter validation;

* - fix import;

* - fix tests;

* - fix tests;

* - ignore tests;

* Feature/certificate revocation load test (#269)

* Feature/certificate revocation load test part 2 (#271)

* - fix bloomfilter input stream read function; (#270)

- remove chunk table;
- fix getSlices db fetch;
- remove unused methods;
- update slice content type to byteArray;
- delete slices when cid not much to remote partition chunks;
- fetch slices by kid,x,y instead of sid;

* Signature hash generation (#273)

* - fix signature hash generation;

* - update bloom filter;

* - fix test;

* Feature/certificate revocation improvements (#272)

* - fix base64Url encoding;
- tar.gz parsing functionality;
- add unit tests;
- minor property changes according to api;

* - remove dead code;

* - bloom filter update;

* - fix tests;

Co-authored-by: oleksandrsarapulovgl <82441124+oleksandrsarapulovgl@users.noreply.github.com>
Co-authored-by: Anže Jenšterle <anze@anze.dev>
oleksandrsarapulovgl pushed a commit that referenced this pull request Feb 6, 2022
@MykhailoNester @oleksandrsarapulovgl
Slices optimization (#263)
7adf6f6 
* - add eTag to all api calls;
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;

* - fix unit test;
oleksandrsarapulovgl added a commit that referenced this pull request Feb 6, 2022
@MykhailoNester @oleksandrsarapulovgl @craftbyte
Api integration (#276)
3732bcb 
* - add revocation module; (#249)

* - add revocation module;

* - update dependency;

* - setup base network logic with use cases; (#250)

* DCC revocation local storage (#253)

* - add interceptor with mock API service responses; (#254)

* Fix parsing of issuer country (#251)

Closes #245

* - add interceptor with mock API service responses;

Co-authored-by: Anže Jenšterle <anze@anze.dev>

* - add etag header; (#255)

- move loading to worker;

* Implemented locate storage for dcc revocation entries (#256)

* - generate 3 types of dcc hashes; (#259)

* Feature/revocation local storage (#258)

* - update network API responses and models; (#257)

- add new usecase;

* Feature/revocation hash lookup (#260)

* Feature/connect api with db (#261)

* Connect API with DB. Update downloading flow;

- add logic for kid loading and storing into DB;
- add API mapper;
- fix base url;
- store last modified in pref.;

* Connect API with DB. Update downloading flow;

- package structure update;
- add new partition structure;
- db update;

* Connect API with DB. Update downloading flow;

- update API model according to new spec;
- update DB columns;
- refactor downloading logic;

* - fix tests;

* - todo item;
- fix test;

* Slices optimization (#263)

* - add eTag to all api calls;
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;

* - fix unit test;

* Slices part; (#265)

* Slices part;

- save slices to DB;
- delete expired data from DB;

* - fix test;

* DB lookup; (#266)

* DB lookup;

- add usecase for hash lookup, mode check and loading data from db;

* - fix tests;

* - add z value to db;
- add enum converter for db;
- update responses with test data;

* Feature/certificate revocation local layer filtering update (#267)

* import bloom filter (#268)

* - import bloom filter with unit tests;
- fix datetime format in DB;
- update KID to base64Url;

* - add bloom filter validation;

* - fix import;

* - fix tests;

* - fix tests;

* - ignore tests;

* Feature/certificate revocation load test (#269)

* Feature/certificate revocation load test part 2 (#271)

* - fix bloomfilter input stream read function; (#270)

- remove chunk table;
- fix getSlices db fetch;
- remove unused methods;
- update slice content type to byteArray;
- delete slices when cid not much to remote partition chunks;
- fetch slices by kid,x,y instead of sid;

* Signature hash generation (#273)

* - fix signature hash generation;

* - update bloom filter;

* - fix test;

* Feature/certificate revocation improvements (#272)

* - fix base64Url encoding;
- tar.gz parsing functionality;
- add unit tests;
- minor property changes according to api;

* - remove dead code;

* - bloom filter update;

* - fix tests;

Co-authored-by: oleksandrsarapulovgl <82441124+oleksandrsarapulovgl@users.noreply.github.com>
Co-authored-by: Anže Jenšterle <anze@anze.dev>
oleksandrsarapulovgl pushed a commit that referenced this pull request Feb 20, 2022
@MykhailoNester @oleksandrsarapulovgl
Slices optimization (#263)
9cb8d67 
* - add eTag to all api calls;
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;

* - fix unit test;
oleksandrsarapulovgl added a commit that referenced this pull request Feb 20, 2022
@MykhailoNester @oleksandrsarapulovgl @craftbyte
Api integration (#276)
1084423 
* - add revocation module; (#249)

* - add revocation module;

* - update dependency;

* - setup base network logic with use cases; (#250)

* DCC revocation local storage (#253)

* - add interceptor with mock API service responses; (#254)

* Fix parsing of issuer country (#251)

Closes #245

* - add interceptor with mock API service responses;

Co-authored-by: Anže Jenšterle <anze@anze.dev>

* - add etag header; (#255)

- move loading to worker;

* Implemented locate storage for dcc revocation entries (#256)

* - generate 3 types of dcc hashes; (#259)

* Feature/revocation local storage (#258)

* - update network API responses and models; (#257)

- add new usecase;

* Feature/revocation hash lookup (#260)

* Feature/connect api with db (#261)

* Connect API with DB. Update downloading flow;

- add logic for kid loading and storing into DB;
- add API mapper;
- fix base url;
- store last modified in pref.;

* Connect API with DB. Update downloading flow;

- package structure update;
- add new partition structure;
- db update;

* Connect API with DB. Update downloading flow;

- update API model according to new spec;
- update DB columns;
- refactor downloading logic;

* - fix tests;

* - todo item;
- fix test;

* Slices optimization (#263)

* - add eTag to all api calls;
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;

* - fix unit test;

* Slices part; (#265)

* Slices part;

- save slices to DB;
- delete expired data from DB;

* - fix test;

* DB lookup; (#266)

* DB lookup;

- add usecase for hash lookup, mode check and loading data from db;

* - fix tests;

* - add z value to db;
- add enum converter for db;
- update responses with test data;

* Feature/certificate revocation local layer filtering update (#267)

* import bloom filter (#268)

* - import bloom filter with unit tests;
- fix datetime format in DB;
- update KID to base64Url;

* - add bloom filter validation;

* - fix import;

* - fix tests;

* - fix tests;

* - ignore tests;

* Feature/certificate revocation load test (#269)

* Feature/certificate revocation load test part 2 (#271)

* - fix bloomfilter input stream read function; (#270)

- remove chunk table;
- fix getSlices db fetch;
- remove unused methods;
- update slice content type to byteArray;
- delete slices when cid not much to remote partition chunks;
- fetch slices by kid,x,y instead of sid;

* Signature hash generation (#273)

* - fix signature hash generation;

* - update bloom filter;

* - fix test;

* Feature/certificate revocation improvements (#272)

* - fix base64Url encoding;
- tar.gz parsing functionality;
- add unit tests;
- minor property changes according to api;

* - remove dead code;

* - bloom filter update;

* - fix tests;

Co-authored-by: oleksandrsarapulovgl <82441124+oleksandrsarapulovgl@users.noreply.github.com>
Co-authored-by: Anže Jenšterle <anze@anze.dev>
oleksandrsarapulovgl added a commit that referenced this pull request Mar 9, 2022
@oleksandrsarapulovgl @MykhailoNester @craftbyte
Feature/certificate revocation (#292)
ec5b567 
* - add revocation module; (#249)

* - add revocation module;

* - update dependency;

* - setup base network logic with use cases; (#250)

* DCC revocation local storage (#253)

* - add interceptor with mock API service responses; (#254)

* Fix parsing of issuer country (#251)

Closes #245

* - add interceptor with mock API service responses;

Co-authored-by: Anže Jenšterle <anze@anze.dev>

* - add etag header; (#255)

- move loading to worker;

* Implemented locate storage for dcc revocation entries (#256)

* - generate 3 types of dcc hashes; (#259)

* Feature/revocation local storage (#258)

* - update network API responses and models; (#257)

- add new usecase;

* Feature/revocation hash lookup (#260)

* Feature/connect api with db (#261)

* Connect API with DB. Update downloading flow;

- add logic for kid loading and storing into DB;
- add API mapper;
- fix base url;
- store last modified in pref.;

* Connect API with DB. Update downloading flow;

- package structure update;
- add new partition structure;
- db update;

* Connect API with DB. Update downloading flow;

- update API model according to new spec;
- update DB columns;
- refactor downloading logic;

* - fix tests;

* - todo item;
- fix test;

* Slices optimization (#263)

* - add eTag to all api calls;
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;

* - fix unit test;

* Slices part; (#265)

* Slices part;

- save slices to DB;
- delete expired data from DB;

* - fix test;

* DB lookup; (#266)

* DB lookup;

- add usecase for hash lookup, mode check and loading data from db;

* - fix tests;

* - add z value to db;
- add enum converter for db;
- update responses with test data;

* Feature/certificate revocation local layer filtering update (#267)

* import bloom filter (#268)

* - import bloom filter with unit tests;
- fix datetime format in DB;
- update KID to base64Url;

* - add bloom filter validation;

* - fix import;

* - fix tests;

* - fix tests;

* - ignore tests;

* Feature/certificate revocation load test (#269)

* Feature/certificate revocation load test part 2 (#271)

* - fix bloomfilter input stream read function; (#270)

- remove chunk table;
- fix getSlices db fetch;
- remove unused methods;
- update slice content type to byteArray;
- delete slices when cid not much to remote partition chunks;
- fetch slices by kid,x,y instead of sid;

* Signature hash generation (#273)

* - fix signature hash generation;

* - update bloom filter;

* - fix test;

* Feature/certificate revocation improvements (#272)

* Api integration (#276)

* - add revocation module; (#249)

* - add revocation module;

* - update dependency;

* - setup base network logic with use cases; (#250)

* DCC revocation local storage (#253)

* - add interceptor with mock API service responses; (#254)

* Fix parsing of issuer country (#251)

Closes #245

* - add interceptor with mock API service responses;

Co-authored-by: Anže Jenšterle <anze@anze.dev>

* - add etag header; (#255)

- move loading to worker;

* Implemented locate storage for dcc revocation entries (#256)

* - generate 3 types of dcc hashes; (#259)

* Feature/revocation local storage (#258)

* - update network API responses and models; (#257)

- add new usecase;

* Feature/revocation hash lookup (#260)

* Feature/connect api with db (#261)

* Connect API with DB. Update downloading flow;

- add logic for kid loading and storing into DB;
- add API mapper;
- fix base url;
- store last modified in pref.;

* Connect API with DB. Update downloading flow;

- package structure update;
- add new partition structure;
- db update;

* Connect API with DB. Update downloading flow;

- update API model according to new spec;
- update DB columns;
- refactor downloading logic;

* - fix tests;

* - todo item;
- fix test;

* Slices optimization (#263)

* - add eTag to all api calls;
- add new API for slices;
- add base logic for 412 error handling;
- add logic for comparing remote chunks slices with local;

* - fix unit test;

* Slices part; (#265)

* Slices part;

- save slices to DB;
- delete expired data from DB;

* - fix test;

* DB lookup; (#266)

* DB lookup;

- add usecase for hash lookup, mode check and loading data from db;

* - fix tests;

* - add z value to db;
- add enum converter for db;
- update responses with test data;

* Feature/certificate revocation local layer filtering update (#267)

* import bloom filter (#268)

* - import bloom filter with unit tests;
- fix datetime format in DB;
- update KID to base64Url;

* - add bloom filter validation;

* - fix import;

* - fix tests;

* - fix tests;

* - ignore tests;

* Feature/certificate revocation load test (#269)

* Feature/certificate revocation load test part 2 (#271)

* - fix bloomfilter input stream read function; (#270)

- remove chunk table;
- fix getSlices db fetch;
- remove unused methods;
- update slice content type to byteArray;
- delete slices when cid not much to remote partition chunks;
- fetch slices by kid,x,y instead of sid;

* Signature hash generation (#273)

* - fix signature hash generation;

* - update bloom filter;

* - fix test;

* Feature/certificate revocation improvements (#272)

* - fix base64Url encoding;
- tar.gz parsing functionality;
- add unit tests;
- minor property changes according to api;

* - remove dead code;

* - bloom filter update;

* - fix tests;

Co-authored-by: oleksandrsarapulovgl <82441124+oleksandrsarapulovgl@users.noreply.github.com>
Co-authored-by: Anže Jenšterle <anze@anze.dev>

* - load chunks from API and store to DB; (#277)

- generate hash based on kid hash type;
- add unit test case for multiple slices gzip parsing;
- check revocation for test/recovery;

* - add retry logic for 412 error case;

* Support for hashlist type; (#278)

* - add new hashlist slice db entity;
- handle hashlist type for slices and store into DB;
- add hashlist type validation during revocation check;
- use 128 bit hash for revocation validation;

* - optimize search by sid list;

* - fix unit test;

* - hashlist testing;

* - enum type update; (#279)

* - add signature check working solution; (#280)

* Fix UCI, CO+UCI hashtype validation (#282)

* - fix uci, co+uci hash check during revocation validation;

* - check certificateRevoked when cose verified;

* Expiration date (#284)

* - store expiration in milliseconds;
- check/delete expired data on app launch;

* - revert to micros;

* Partial-variable length revocation data encoding (#287)

* Api optimization (#286)

* - add new API to fetch chunks for partition;

* - add new API to fetch chunk slices by sids;

* - remove empty lines;

* Implemented partial variable length hashes fill up (#290)

Co-authored-by: Mykhailo Nester <nsmisha.dev@gmail.com>
Co-authored-by: Anže Jenšterle <anze@anze.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oleksandrsarapulovgl oleksandrsarapulovgl Awaiting requested review from oleksandrsarapulovgl

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@MykhailoNester