Improve publisher list insert time/memory/cpu

[NejcZdovc]

We have a big performance problem on iOS when we are inserting publisher list into db (we need to insert over 600k publishers). We need to review code and fix memory and CPU problems for this process.

[kjozwiak]

Verification PASSED on macOS 10.15.4 x64 using the following build:

Brave | 1.8.84 Chromium: 81.0.4044.122 (Official Build) (64-bit)
-- | --
Revision | 44f4233f08910d83b146130c1938256a2e05b136-refs/branch-heads/4044@{#963}
OS | macOS Version 10.15.4 (Build 19E287)

• Went through the cases outlined via Publisher list improvements brave-core#5301 (comment) and Publisher list improvements brave-core#5301 (comment).

Verified that the Publisher List was being downloaded without any issues once brave://rewards was enabled:

[ RESPONSE - OnDownload ]
> time: 1588029200
> result: Success
> http code: 200
> response: Publisher list
> headers accept-ranges: bytes
> headers access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
> headers access-control-allow-methods: GET
> headers access-control-allow-origin: https://rewards.bravesoftware.com
> headers access-control-request-method: GET
> headers age: 564
> headers cache-control: max-age=3600, public
> headers content-encoding: gzip
> headers content-length: 19006799
> headers content-type: application/json; charset=utf-8
> headers date: Mon, 27 Apr 2020 22:57:51 GMT
> headers etag: W/"4b8454a1c57a8bdf4f924f70507b812c"
> headers server: Cowboy
> headers status: 200
> headers vary: Accept-Encoding
> headers via: 1.1 varnish
> headers x-cache: HIT
> headers x-cache-hits: 0
> headers x-frame-options: deny
> headers x-request-id: 79bb2085-eaf8-4293-adc2-6cbbf83afd2c
> headers x-runtime: 2.669314
> headers x-served-by: cache-yyz4545-YYZ
[ END RESPONSE ]

Ensured that brave.com, chess.com and kjozwiak.github.com appeared as verified and displayed the custom banners without any issues.

brave.com example:

chess.com example:

kjozwiak.github.io example:

Also checked and ensured that the server_publisher_info database was populated as per the following:

Ensured that publisher_info was populated whenever you visited a website:

Verification passed on

Brave	1.8.85 Chromium: 81.0.4044.122 (Official Build) (64-bit)
Revision	44f4233f08910d83b146130c1938256a2e05b136-refs/branch-heads/4044@{#963}
OS	Windows 10 OS Version 1803 (Build 17134.1006)

• Went through the cases outlined via Publisher list improvements brave-core#5301 (comment) and Publisher list improvements brave-core#5301 (comment).

Verified that the Publisher List was being downloaded without any issues once brave://rewards was enabled:

[ RESPONSE - OnDownload ]
> time: 1588073906
> result: Success
> http code: 200
> response: Publisher list
> headers accept-ranges: bytes
> headers access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
> headers access-control-allow-methods: GET
> headers access-control-allow-origin: https://rewards.bravesoftware.com
> headers access-control-request-method: GET
> headers age: 358
> headers cache-control: max-age=3600, public
> headers content-encoding: gzip
> headers content-length: 19033260
> headers content-type: application/json; charset=utf-8
> headers date: Tue, 28 Apr 2020 11:34:35 GMT
> headers etag: W/"badde5f9b5a50b7a9ae683a56cb877d5"
> headers server: Cowboy
> headers status: 200
> headers vary: Accept-Encoding
> headers via: 1.1 varnish
> headers x-cache: HIT
> headers x-cache-hits: 30
> headers x-frame-options: deny
> headers x-request-id: 6cf80531-c397-45e4-b7f9-7428a3cb3d44
> headers x-runtime: 2.077662
> headers x-served-by: cache-bom18226-BOM
[ END RESPONSE ]

Ensured that brianbondy.com, clifton.io and duckduckgo.com appeared as verified and displayed the custom banners without any issues.

Also checked and ensured that the server_publisher_info database was populated as per the following:

Ensured that publisher_info was populated whenever you visited a website:

Verification passed on

Brave	1.8.86 Chromium: 81.0.4044.129 (Official Build) (64-bit)
Revision	3d71af9f5704a40b85806f4d08925db24605ba25-refs/branch-heads/4044@{#979}
OS	Ubuntu 18.04 LTS

• Went through the cases outlined via Publisher list improvements brave-core#5301 (comment) and Publisher list improvements brave-core#5301 (comment).

Verified that the Publisher List was being downloaded without any issues once brave://rewards was enabled:

[ RESPONSE - OnDownload ]
> time: 1588171026
> result: Success
> http code: 200
> response: Publisher list
> headers accept-ranges: bytes
> headers access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
> headers access-control-allow-methods: GET
> headers access-control-allow-origin: https://rewards.bravesoftware.com
> headers access-control-request-method: GET
> headers age: 876
> headers cache-control: max-age=3600, public
> headers content-encoding: gzip
> headers content-length: 19170584
> headers content-type: application/json; charset=utf-8
> headers date: Wed, 29 Apr 2020 14:37:03 GMT
> headers etag: W/"28c9c2762e07b939d5068859fac025bf"
> headers server: Cowboy
> headers status: 200
> headers strict-transport-security: max-age=31536000; includeSubDomains
> headers vary: Accept-Encoding
> headers via: 1.1 varnish
> headers x-cache: HIT
> headers x-cache-hits: 0
> headers x-frame-options: deny
> headers x-request-id: 62f1554b-4b69-454a-bd35-2c1532977f3c
> headers x-runtime: 4.994960
> headers x-served-by: cache-hhn4065-HHN
[ END RESPONSE ]

Ensured that brave.com appeared as verified and displayed the custom banners without any issues.

brave.com example:

Also checked and ensured that the server_publisher_info database was populated as per the following:

Ensured that publisher_info was populated whenever you visited a website: