Skip to content

Commit

Permalink
For mozilla-mobile#12184 - Add support for pacing and rotating sponso…
Browse files Browse the repository at this point in the history
…red Pocket stories

Stories can be paced and rotated based on the properties received in the
endpoint response.

Since rotating involves a limit of impressions in a certain period I've added
a new table for keeping only this timestamps while the effective limits will be
held in the sponsored stories table.

With very little time between this and the previous patch which added support
for sponsored stories I've skipped created a new database version and sticked
to using version 2 again to ensure a smoother migration when this feature gets
to the users.

Possibly because of the foreignKey addition the migration could not be tested
in the JVM (because of sqlite exceptions coming from robolectric) and so I
switched testing this to a real device.
  • Loading branch information
Mugurell committed May 18, 2022
1 parent 0c5e572 commit c07480f
Show file tree
Hide file tree
Showing 29 changed files with 1,408 additions and 190 deletions.
26 changes: 17 additions & 9 deletions components/service/pocket/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,25 @@ Currently this supports:

## Usage
1. For Pocket recommended stories:
- Use `PocketStoriesService#startPeriodicStoriesRefresh` and `PocketStoriesService#stopPeriodicStoriesRefresh`
as high up in the client app as possible (preferably in the Application object or in a single Activity) to ensure the
background story refresh functionality works for the entirety of the app lifetime.
- Use `PocketStoriesService.getStories` to get the current list of Pocket recommended stories.
- Use `PocketStoriesService#startPeriodicStoriesRefresh` and `PocketStoriesService#stopPeriodicStoriesRefresh`
as high up in the client app as possible (preferably in the Application object or in a single Activity) to ensure the
background story refresh functionality works for the entirety of the app lifetime.
- Use `PocketStoriesService.getStories` to get the current list of Pocket recommended stories.

2. For Pocket sponsored stories:
- Use `PocketStoriesService#startPeriodicSponsoredStoriesRefresh` and `PocketStoriesService#stopPeriodicSponsoredStoriesRefresh`
as high up in the client app as possible (preferably in the Application object or in a single Activity) to ensure the
background story refresh functionality works for the entirety of the app lifetime.
- Use `PocketStoriesService.getSponsoredStories` to get the current list of Pocket recommended stories.
- Use `PocketStoriesService.deleteProfile` to delete all server stored information about the device to which sponsored stories were previously downloaded. This may include data like network ip and application tokens.
- Use `PocketStoriesService#startPeriodicSponsoredStoriesRefresh` and `PocketStoriesService#stopPeriodicSponsoredStoriesRefresh`
as high up in the client app as possible (preferably in the Application object or in a single Activity) to ensure the
background story refresh functionality works for the entirety of the app lifetime.
- Use `PocketStoriesService.getSponsoredStories` to get the current list of Pocket recommended stories.
- Use `PocketStoriesService,recordStoriesImpressions` to try and persist that a list of sponsored stories were shown to the user. (Safe to call even if those stories are not persisted).
- Use `PocketStoriesService.deleteProfile` to delete all server stored information about the device to which sponsored stories were previously downloaded. This may include data like network ip and application tokens.

##### Pacing and rotating:
A new `PocketSponsoredStoryCaps` is available in the response from `PocketStoriesService.getSponsoredStories` which allows checking `currentImpressions`, `lifetimeCount`, `flightCount`, `flightPeriod` based on which the client can decide which stories to show.
All this is based on clients calling `PocketStoriesService,recordStoriesImpressions` to record new impressions in between application restarts.




### Setting up the dependency

Expand Down
10 changes: 10 additions & 0 deletions components/service/pocket/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ android {
defaultConfig {
minSdkVersion config.minSdkVersion
targetSdkVersion config.targetSdkVersion
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"

kapt {
arguments {
Expand All @@ -29,6 +30,7 @@ android {

sourceSets {
test.assets.srcDirs += files("$projectDir/schemas".toString())
androidTest.assets.srcDirs += files("$projectDir/schemas".toString())
}
}

Expand Down Expand Up @@ -58,6 +60,14 @@ dependencies {

testImplementation project(':support-test')
testImplementation project(':lib-fetch-httpurlconnection')

androidTestImplementation project(':support-android-test')

androidTestImplementation Dependencies.androidx_room_testing
androidTestImplementation Dependencies.androidx_arch_core_testing
androidTestImplementation Dependencies.androidx_test_core
androidTestImplementation Dependencies.androidx_test_runner
androidTestImplementation Dependencies.androidx_test_rules
}

apply from: '../../../publish.gradle'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"formatVersion": 1,
"database": {
"version": 2,
"identityHash": "1ea41b5cc0791d92dd8f0db8b387fe6c",
"identityHash": "966f55824415a21a73640bd2641772f2",
"entities": [
{
"tableName": "stories",
Expand Down Expand Up @@ -62,8 +62,14 @@
},
{
"tableName": "spocs",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`url` TEXT NOT NULL, `title` TEXT NOT NULL, `imageUrl` TEXT NOT NULL, `sponsor` TEXT NOT NULL, `clickShim` TEXT NOT NULL, `impressionShim` TEXT NOT NULL, PRIMARY KEY(`url`))",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` INTEGER NOT NULL, `url` TEXT NOT NULL, `title` TEXT NOT NULL, `imageUrl` TEXT NOT NULL, `sponsor` TEXT NOT NULL, `clickShim` TEXT NOT NULL, `impressionShim` TEXT NOT NULL, `priority` INTEGER NOT NULL, `lifetimeCapCount` INTEGER NOT NULL, `flightCapCount` INTEGER NOT NULL, `flightCapPeriod` INTEGER NOT NULL, PRIMARY KEY(`id`))",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "url",
"columnName": "url",
Expand Down Expand Up @@ -99,22 +105,90 @@
"columnName": "impressionShim",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "priority",
"columnName": "priority",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "lifetimeCapCount",
"columnName": "lifetimeCapCount",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "flightCapCount",
"columnName": "flightCapCount",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "flightCapPeriod",
"columnName": "flightCapPeriod",
"affinity": "INTEGER",
"notNull": true
}
],
"primaryKey": {
"columnNames": [
"url"
"id"
],
"autoGenerate": false
},
"indices": [],
"foreignKeys": []
},
{
"tableName": "spocs_impressions",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`spocId` INTEGER NOT NULL, `impressionId` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, `impressionDateInSeconds` INTEGER NOT NULL, FOREIGN KEY(`spocId`) REFERENCES `spocs`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )",
"fields": [
{
"fieldPath": "spocId",
"columnName": "spocId",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "impressionId",
"columnName": "impressionId",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "impressionDateInSeconds",
"columnName": "impressionDateInSeconds",
"affinity": "INTEGER",
"notNull": true
}
],
"primaryKey": {
"columnNames": [
"impressionId"
],
"autoGenerate": true
},
"indices": [],
"foreignKeys": [
{
"table": "spocs",
"onDelete": "CASCADE",
"onUpdate": "NO ACTION",
"columns": [
"spocId"
],
"referencedColumns": [
"id"
]
}
]
}
],
"views": [],
"setupQueries": [
"CREATE TABLE IF NOT EXISTS room_master_table (id INTEGER PRIMARY KEY,identity_hash TEXT)",
"INSERT OR REPLACE INTO room_master_table (id,identity_hash) VALUES(42, '1ea41b5cc0791d92dd8f0db8b387fe6c')"
"INSERT OR REPLACE INTO room_master_table (id,identity_hash) VALUES(42, '966f55824415a21a73640bd2641772f2')"
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */

package mozilla.components.service.pocket.stories.db

import android.content.Context
import androidx.arch.core.executor.testing.InstantTaskExecutorRule
import androidx.room.Room
import androidx.room.testing.MigrationTestHelper
import androidx.sqlite.db.framework.FrameworkSQLiteOpenHelperFactory
import androidx.test.core.app.ApplicationProvider
import androidx.test.platform.app.InstrumentationRegistry
import kotlinx.coroutines.runBlocking
import mozilla.components.service.pocket.spocs.db.SpocEntity
import mozilla.components.service.pocket.spocs.db.SpocImpressionEntity
import mozilla.components.service.pocket.stories.db.PocketRecommendationsDatabase.Companion
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors

private const val MIGRATION_TEST_DB = "migration-test"

class PocketRecommendationsDatabaseTest {
private lateinit var context: Context
private lateinit var executor: ExecutorService
private lateinit var database: PocketRecommendationsDatabase

@get:Rule
@Suppress("DEPRECATION")
val helper: MigrationTestHelper = MigrationTestHelper(
InstrumentationRegistry.getInstrumentation(),
PocketRecommendationsDatabase::class.java.canonicalName,
FrameworkSQLiteOpenHelperFactory()
)

@get:Rule
var instantTaskExecutorRule = InstantTaskExecutorRule()

@Before
fun setUp() {
executor = Executors.newSingleThreadExecutor()

context = ApplicationProvider.getApplicationContext()
database = Room.inMemoryDatabaseBuilder(context, PocketRecommendationsDatabase::class.java).build()
}

@After
fun tearDown() {
executor.shutdown()
database.close()
}

@Test
fun `test1To2MigrationAddsNewSpocsAndSpocsImpressionsTables`() = runBlocking {
// Create the database with the version 1 schema
val dbVersion1 = helper.createDatabase(MIGRATION_TEST_DB, 1).apply {
execSQL(
"INSERT INTO " +
"'${Companion.TABLE_NAME_STORIES}' " +
"(url, title, imageUrl, publisher, category, timeToRead, timesShown) " +
"VALUES (" +
"'${story.url}'," +
"'${story.title}'," +
"'${story.imageUrl}'," +
"'${story.publisher}'," +
"'${story.category}'," +
"'${story.timeToRead}'," +
"'${story.timesShown}'" +
")"
)
}
// Validate the persisted data which will be re-checked after migration
dbVersion1.query(
"SELECT * FROM ${Companion.TABLE_NAME_STORIES}"
).use { cursor ->
assertEquals(1, cursor.count)

cursor.moveToFirst()
assertEquals(
story,
PocketStoryEntity(
url = cursor.getString(0),
title = cursor.getString(1),
imageUrl = cursor.getString(2),
publisher = cursor.getString(3),
category = cursor.getString(4),
timeToRead = cursor.getInt(5),
timesShown = cursor.getLong(6),
)
)
}

val impression = SpocImpressionEntity(spoc.id).apply {
impressionId = 1
impressionDateInSeconds = 700L
}
// Migrate the initial database to the version 2 schema
val dbVersion2 = helper.runMigrationsAndValidate(
MIGRATION_TEST_DB, 2, true, Migrations.migration_1_2
).apply {
execSQL(
"INSERT INTO " +
"'${Companion.TABLE_NAME_SPOCS}' (" +
"id, url, title, imageUrl, sponsor, clickShim, impressionShim, " +
"priority, lifetimeCapCount, flightCapCount, flightCapPeriod" +
") VALUES (" +
"'${spoc.id}'," +
"'${spoc.url}'," +
"'${spoc.title}'," +
"'${spoc.imageUrl}'," +
"'${spoc.sponsor}'," +
"'${spoc.clickShim}'," +
"'${spoc.impressionShim}'," +
"'${spoc.priority}'," +
"'${spoc.lifetimeCapCount}'," +
"'${spoc.flightCapCount}'," +
"'${spoc.flightCapPeriod}'" +
")"
)

execSQL(
"INSERT INTO " +
"'${Companion.TABLE_NAME_SPOCS_IMPRESSIONS}' (" +
"spocId, impressionId, impressionDateInSeconds" +
") VALUES (" +
"'${impression.spocId}'," +
"'${impression.impressionId}'," +
"'${impression.impressionDateInSeconds}'" +
")"
)
}
// Re-check the initial data we had
dbVersion2.query(
"SELECT * FROM ${Companion.TABLE_NAME_STORIES}"
).use { cursor ->
assertEquals(1, cursor.count)

cursor.moveToFirst()
assertEquals(
story,
PocketStoryEntity(
url = cursor.getString(0),
title = cursor.getString(1),
imageUrl = cursor.getString(2),
publisher = cursor.getString(3),
category = cursor.getString(4),
timeToRead = cursor.getInt(5),
timesShown = cursor.getLong(6),
)
)
}
// Finally validate that the new spocs are persisted successfully
dbVersion2.query(
"SELECT * FROM ${Companion.TABLE_NAME_SPOCS}"
).use { cursor ->
assertEquals(1, cursor.count)

cursor.moveToFirst()
assertEquals(spoc.id, cursor.getInt(0))
assertEquals(spoc.url, cursor.getString(1))
assertEquals(spoc.title, cursor.getString(2))
assertEquals(spoc.imageUrl, cursor.getString(3))
assertEquals(spoc.sponsor, cursor.getString(4))
assertEquals(spoc.clickShim, cursor.getString(5))
assertEquals(spoc.impressionShim, cursor.getString(6))
assertEquals(spoc.priority, cursor.getInt(7))
assertEquals(spoc.lifetimeCapCount, cursor.getInt(8))
assertEquals(spoc.flightCapCount, cursor.getInt(9))
assertEquals(spoc.flightCapPeriod, cursor.getInt(10))
}
// And that the impression was also persisted successfully
dbVersion2.query(
"SELECT * FROM ${Companion.TABLE_NAME_SPOCS_IMPRESSIONS}"
).use { cursor ->
assertEquals(1, cursor.count)

cursor.moveToFirst()
assertEquals(impression.spocId, cursor.getInt(0))
assertEquals(impression.impressionId, cursor.getInt(1))
assertEquals(impression.impressionDateInSeconds, cursor.getLong(2))
}
}
}

private val story = PocketStoryEntity(
title = "How to Get Rid of Black Mold Naturally",
url = "https://getpocket.com/explore/item/how-to-get-rid-of-black-mold-naturally",
imageUrl = "https://img-getpocket.cdn.mozilla.net/{wh}/filters:format(jpeg):quality(60):no_upscale():strip_exif()/https%3A%2F%2Fpocket-image-cache.com%2F1200x%2Ffilters%3Aformat(jpg)%3Aextract_focal()%2Fhttps%253A%252F%252Fpocket-syndicated-images.s3.amazonaws.com%252Farticles%252F6757%252F1628024495_6109ae86db6cc.png",
publisher = "Pocket",
category = "general",
timeToRead = 4,
timesShown = 23
)

private val spoc = SpocEntity(
id = 191739319,
url = "https://i.geistm.com/l/GC_7ReasonsKetoV2_Journiest?bcid=601c567ac5b18a0414cce1d4&bhid=624f3ea9adad7604086ac6b3&utm_content=PKT_A_7ReasonsKetoV2_Journiest_40702022_RawMeatballUGC_130Off_601c567ac5b18a0414cce1d4_624f3ea9adad7604086ac6b3&tv=su4&ct=NAT-PK-PROS-130OFF5WEEK-037&utm_medium=DB&utm_source=pocket~geistm&utm_campaign=PKT_A_7ReasonsKetoV2_Journiest_40702022_RawMeatballUGC_130Off",
title = "Eating Keto Has Never Been So Easy With Green Chef",
imageUrl = "https://img-getpocket.cdn.mozilla.net/direct?url=realUrl.png&resize=w618-h310",
sponsor = "Green Chef",
clickShim = "193815086ClickShim",
impressionShim = "193815086ImpressionShim",
priority = 3,
lifetimeCapCount = 50,
flightCapCount = 10,
flightCapPeriod = 86400,
)
Loading

0 comments on commit c07480f

Please sign in to comment.