Detail View

With the advent of mobile computing and communication devices, the Experience Sampling Method (ESM) has been increasingly used as a convenient and effective approach to evaluate mobile and context-dependent applications and capture their users’ ecological behaviors. 
Similar to any field-based in situ testing methods, the ESM is prone to biases from unreliable and unbalanced data, also A lot of economical cost is required for operating experiments and data analysis, especially for A/B testing and context-specific situations. In fact, ESM has rarely been applied to A/B testing, and literatures revealing its operational details and potential difficulties are very rare. 
In this thesis, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. 
And then, we present an enhanced ESM (and an associated tool) called Parameter REconfiguring experience Sampling and analysis TOol (PRESTO), which has the capability to dynamically reconfigure evaluation parameters to help manage an experiment to be more reliable, economical, and convenient. 
With PRESTO, the experimenter can continuously analyze the incoming data for their sufficiency, balance, power (of the test) and reliability, and thereby, e.g. shorten or pay more attention on certain segments of the experiment. To validate its effectiveness, a case study of applying PRESTO to the A/B testing and evaluation of two competing interfaces for a mobile application, as compared to using a conventional ESM, was carried out. Based on the gathered data and direct interviews with the participants, we demonstrate how it could help manage ESM-based A/B testing more efficiently and reliably, and even derive different outcomes.

Abstract………………………………………………………………………………...iAcknowledgementsiiiCONTENTSvList of FiguresviiiList of TablesxiiChapter 1.Introduction11.1 Motivation11.2 Experience Sampling Method/Tools and Its Evolution61.3 Potential Problems with Experience Sampling Method9Chapter 2.On Applying Experience Sampling Method to A/B Testing of Mobile Applications: A Case Study142.1 Test Application and Evaluated Interfaces142.2 ESM based A/B Testing Process and the Support Tool152.3 Detailed Experiment Procedure172.3.1 Participants and Experiment Duration182.3.2 Data Collection Schedule and Alarms182.3.3 Survey, Contextual, and Gold-Standard Questions192.3.4 User Behavior212.3.5 Compensation212.3.6 Questionnaires about the ESM procedure itself.212.3.7 Data collected from participant232.4 Data Sufficiency and Balance242.5 Participant Reliability262.6 Experiment Extension and the Utility of ESM272.7 Participant Responses about the ESM Process Itself292.8 Identified issues31Chapter 3.PRESTO: Experience Sampling with On-line Reconfiguration of Experimental Parameters for Evaluation of Mobile Applications: An Overview333.1 Intermediate analysis and evolved ESM process333.2 PRESTO Architecture37Chapter 4.An Overview Comparison Study: Conventional ESM vs. PRESTO for A/B Testing424.1 Type of A/B Testing424.2 Test Application and Evaluated Interfaces424.3 Detailed Experiment Procedure434.3.1 Participants and Experiment Duration464.3.2 Data Collection Schedule and Alarms484.3.3 Survey, Contextual, and Gold Standard Questions514.3.4 User Behavior524.4 Intermediate Analysis and Parameter Reconfiguration (PRESTO Only)554.4.1 Data Sufficiency554.4.2 Data/Subject Filtering564.4.3 Analysis Method584.4.4 Analysis Frequency and Rescheduling594.5 Results and Observations614.5.1 Data Sufficiency and Balance – Non-PRESTO614.5.2 Data Sufficiency and Balance – PRESTO664.5.3 Participant Reliability704.5.4 Economy and Utility of PRESTO734.5.5 Actual Experimental Results74Chapter 5.DISCUSSION78Chapter 6.CONCLUSION AND FUTURE WORK80References………………………………………………………………………………81