Detailed Test Methodology Normal Benchmarks

The detailed steps we used to perform the benchmarks discussed in "Section 3 - JSON vs. XML" of the Performance Bencmarking page are listed below:

IMPORTANT - Note that Step #9 below, which clears the existing process from memory to avoid back-to-back comparisons in the same app execution (i.e., warm starts), is extremely important to ensure fair test results across different configurations.

1. Install the SiriRestClientUI from the APK (i.e., not installed via Eclipse deploy process). No obfuscation is used in the current app version.
2. Connect your device to computer with Android DDMS running so you can view Logcat output.
3. Start the app
4. Set appropriate settings for the test via the 'Settings" menu item (e.g., Parsing XML vs. JSON, internal vs. Android HTTP connection, number of consecutive tests, time between tests, beep on test complete, etc.)
5. Browse to the "Stop_Request" tab in the app.
6. Use the following parameters, and then press the 'Submit" button:

• key=add-your-key-here
• OperatorRef=MTA
• MonitoringRef=308214
• StopMonitoringDetailLevel=normal
• MaximumNumberOfCallsOnwards=1

7. Check the LogCat output. If you executed just one test consecutively, you should see this "elapsed time" result at the end of the test. You can just copy/paste this into Excel file, and proceed to Step #9. However, if you execute multiple tests, when all tests are complete, you should see a comma-delimited list of elapsed response times in the log. Click on this, and copy and paste this into a new .CSV file that you create. You'll need to remove the beginning timestamp and info from the log, as well as the trailing bracket. For example, you'll paste the following text - 01-08 12:18:17.133: D/SiriRestClientUI(7957): [472.485884, 4981.10789, 5076.606135, 5520.280787, 5355.745459, 5243.094766, 5109.629177, 5228.170304, 6932.275292, 5249.198839, 4993.987486, 5397.466809, 5602.258507, 5281.977721, 7573.355714, 6665.862963, 6040.622616, 5530.230429, 5260.857622, 5797.375249] - but you should change it so its just the comma-delimited times, such as 472.485884, 4981.10789, 5076.606135, 5520.280787, 5355.745459, 5243.094766, 5109.629177, 5228.170304, 6932.275292, 5249.198839, 4993.987486, 5397.466809, 5602.258507, 5281.977721, 7573.355714, 6665.862963, 6040.622616, 5530.230429, 5260.857622, 5797.375249. Save this .CSV file with a name such as test1.csv.
8. Open the CSV file in Excel. You should see the times each in their own row. Copy these values, then open the main spreadsheet you're using to track elapsed response times and click on the top cell of the column you want to paste the values in. Then, do Paste->Paste Special and check the Transpose option, and click Ok. This will give you a column of the response times that you can use to create a graph or do other analysis, next to columns from other tests you will run.
9. Before switching to test another configuration (e.g., testing JSON instead of XML) and repeating Steps #3-8, please be sure to exit the app (i.e., press the "Back" button), and go to the main phone 'Settings->Application Manager' section. Here, there are typically at least two lists of apps/processes: "Running", and "Cached". To ensure fair performance comparisons due to cold vs. warm start differences (discussed in the following sections), you want to remove the app from both the "Running" and "Cached" lists. If you properly exited the app using the "Back" button, the app should already be removed from the "Running" list. For example, to access the "Cached" list on the Samsung Galaxy S3, you need to click on the "Show cached processes" button in the header of the "Running" application list. Then, tap on "Siri Rest Client" in the list of cached processes, and then tap on "Stop" to remove it from the cached process list. This ensures that the next application startup will be a cold start, which is a fair comparison against the initial startup of the app after installation, the state after a device reboot, and the state after the Android system automatically removes the app from the cached processes list (e.g., when memory is running low and you're using other applications).