Detailed Test Methodology Cold Start Benchmarks

The detailed steps performed for the Section 3 - JSON vs. XML cold start tests from the Performance Benchmarks page are listed below:

1. Install the SiriRestClientUI from the APK (i.e., not installed via Eclipse deploy process). No obfuscation is used in the current app version.
2. Connect your device to computer with Android DDMS running so you can view Logcat output.
3. Start the app
4. Make sure in "Settings->Cache Jackson Objects" is set to false, and that "Number of consecutive requests" is set to 1.
5. Browse to the "Stop_Request" tab in the app.
6. Use the following parameters, and then press the "Submit" button:

• key=add-your-key-here
• OperatorRef=MTA
• MonitoringRef=308214
• StopMonitoringDetailLevel=normal
• MaximumNumberOfCallsOnwards=1

7. Check the LogCat output. You should see this "Elapsed Time" result at the end of the test. You can just copy/paste this into Excel file as the "Cold start time".
8. Set "Settings->Cache Jackson Objects" is set to true, and press the "Submit" button again. Check the LogCat output. You should see a message Wrote ObjectReader.cache to cache (226526 bytes) in 2,924.493 ms. Record the time value as the "Time to write cache", and the number of bytes as the "Size of Cache."
9. Exit the app (i.e., press the "Back" button), and go to the main phone 'Settings->Application Manager' section. Here, there are typically at least two lists of apps/processes: "Running", and "Cached". For example, to access the "Cached" list on the Samsung Galaxy S3, you need to click on the "Show cached processes" button in the header of the "Running" application list. Then, tap on "Siri Rest Client" in the list of cached processes, and then tap on "Stop" to remove it from the cached process list. Hit the "Back" button and wait for the Siri Rest Client" to be removed from the "Cached Process List"
10. Start up the app again, and immediately look at the LogCat output. The Siri Rest Client should immediately read the cached object on app startup, and you should see a message Read ObjectReader.cache from cache (226526 bytes) in 3,505.692 ms. Record the time value as the "Time to read cache", and the number of bytes should match the previous cache write message.
11. Repeat Steps #5 and #6 to make another request to the server, which will create a pseudo-warm start request using the cached Jackson object. Then, check the LogCat output. You should see this "Elapsed Time" result at the end of the test. You can just copy/paste this into Excel file as the "Elapsed Response Time (Using Cached Object)".
12. After a few seconds delay, you should also see another cache write message, where the most recent Jackson object is being cached again. Compare the size of bytes of this cache write with the size recorded in Step #8. If the sizes are different, the contents of the SIRI response may have changed. Record this observation and new byte size in the "Notes" field of the spreadsheet.
13. The spreadsheet will automatically calculate the "Total Pseudo-Warm Start Time" (based on the "Time to read cache" + the "Elapsed Response Time (Using Cached Object)" values), as well as the difference between the "Total Pseudo-Warm Start Time" and the cold start time in both milliseconds and percent difference ((Cold-PseudoWarm)/Cold).
14. To reset for the next round of tests, perform Step #4 (Turning off caching) and then Step #9 (clearing the cached process from memory). Then, start the process over again by beginning at Step #3.