Social and Organizational Impacts of Emerging Mobile Devices: Evaluating Use

2010 saw the International Journal of Mobile Human Computer Interaction (IJMHCI) celebrate its second year in print – two years in which we have seen innovation, high quality research, and thought provoking and challenging articles come together to create exciting issues of the journal.

This Advances book is a compendium of articles from the second volume of the International Journal of Mobile Human Computer Interaction (IJMHCI). The mission of the IJMHCI is to provide an international forum for researchers, educators, and practitioners to advance knowledge and practice in all facets of design and evaluation of human interaction with mobile technologies; to encourage readers to think out of the box to ensure that novel, effective user interface design and evaluation strategies continue to emerge and, in turn, the true potential of mobile technology is realized whilst being sensitive to the societal impact such technologies may have. The IJMHCI brings together a comprehensive collection of research articles from international experts on the design, evaluation, and use of innovative handheld, mobile, and wearable technologies; it also considers issues associated with the social and/or organizational impacts of such technologies. Emerging theories, methods, and interaction designs are included and are complemented with case studies which demonstrate the practical application of these new ideas. The aim of the journal is to increase exposure to, and heighten awareness of, the complexity of current and future issues concerning mobile human-computer interaction. In its fledgling years, and over the life of the journal, articles have presented (and will undoubtedly and indeed hopefully continue to present) alternative points of view for some of the field’s hotly debated topics. Such variance is not only stimulating but also essential in terms of encouraging readers to think to the future and embrace the challenge of new paradigms both for interaction design and evaluation.

The second volume of the IJMHCI comprised 4 issues which, collectively, highlight the vibrant nature of the field of mobile HCI and showcase some of the interesting and exciting work that is being done. In large part, a broader societal focus emerged from the articles in this volume: from design for children to the use of mobile technology in the context of the real world, we saw articles covering a diverse range of innovation, issues and concepts.

The first three articles included in this publication appeared in issue 2(1) of the IJMHCI. Interesting and comprehensive yet diverse articles, they exemplify the broad spectrum of research encompassed by the field of Mobile HCI and thereby captured within the journal. In the first article, entitled “On Quality of Experience in Remote Visualization on Mobile Devices”, Gianluca Paravati, Andrea Sanna, Fabrizio Lamberti, and Luigi Ciminiera introduce the concept of Quality of Experience (QoE) as a relatively novel means by which to measure user satisfaction in the use of a mobile service. In particular, they look at QoE associated with manipulating 3D scenes on mobile devices – visualization of complex 3D objects on mobile devices being increasingly plausible and useful for field engineers, maintenance engineers, etc. The authors introduce a novel approach for managing the properties of video which is streamed between a rendering server and a remote (mobile) client and which permits parameters such as frame resolution, frame rate, and image quality to be tuned according to both user preferences and network performance. Based on experimental evidence, the authors discuss the relationship between these tailorable parameters and user perception of QoE.

The second article, written by Katrin Arning and Martina Ziefle, is entitled “Ask and You Will Receive: Training Novice Adults to Use a PDA in an Active Learning Environment”. Katrin and Martina assert that mobile devices have predominantly been designed to suit the demands, knowledge, and cognitive abilities of technology-savvy and younger users and, as such, neglect the specific needs of older adult users and/or novice adult users. In this article, Katrin and Martina explore the effect of asking questions and repeated practice on PDA skill teaching and acquisition in older adults; additionally, they investigate information deficits and shortcomings in the older adults’ mental models as well as the effectiveness of an active learning environment for older users’ successful PDA skill acquisition. They note that although asking questions and exposure to repeated practice improved the older adults’ PDA performance, such protocols did not enhance the users’ perception of the technology’s ease of use. Interestingly, by exploring the questions asked by the adult learners, Katrin and Martina are able to illustrate the information needs, cognitive barriers, and mental models of novice adult learners and posit this as information which can inform the design of future interfaces to mobile applications.

In the third article, entitled “A Model-Based Approach to Analysis and Calibration of Sensor-Based Human Interaction Loops”, Parisa Eslambolchilar and Roderick Murray-Smith introduce us to manual control theory as it applies to analysis of the continuous aspects of interaction design and human behavior with respect to mobile technologies. The authors argue that current approaches to the design of mobile applications are typically based on trial and error and, as such, would benefit from a formal model which is able to predict the interaction process between the human user and the application. To this end, they present a theoretical framework for specifying, analyzing, and calibrating a sensor-based scrolling and zooming application for mobile devices whilst including the user in the interaction loop. The authors note that this approach is especially timely and useful for guiding the design of sensor-based mobile applications. Parisa and Roderick test their framework on a tilt-controlled, speed-dependent, automatic zooming application on a PDA.

Directing our attention to mobile interaction design as it applies to children, issue 2(2) of the IJMHCI was a special issue on Mobile Interaction Design & Children guest-edited by esteemed researchers in the field, Janet C. Read, Panos Markopoulos, and Allison Druin. As the guest editors noted, mobile technology and children are a natural pairing. Although there have been workshops and books devoted to this topic, Vol. 2, Issue 2 of the IJMHCI represented the first ever dedicated collection of academic titles to appear on this topic within a journal. Interested readers should refer to IJMHCI Vol 2., Issue 2 to read the guest editors’ preface which clearly outlines the motivation behind the special issue; for now, we draw on their introduction to each of the articles in this fantastic collection in order to entice readers into the world of mobile design for children – to quote the guest editors: “the technologies that children will use in ten years time have already been imagined, but what these children will do with these technologies cannot be imagined! That is why studying ‘children and their interactions with mobile technologies’ is so fascinating and so rewarding. Read the enthusiasm in the papers presented here, catch that enthusiasm and enter into our world”.

Acknowledging the difficulties associated with studying interactions with children and mobile technologies, the first of the articles in the collection – “Children’s Interaction with Mobile Touch-Screen Devices: Experiences and Guidelines for Design” by Lorna McKnight and Brendan Cassidy – describes empirical studies carried out with children using a touchscreen Nokia Internet tablet. Providing insight into how to conduct studies of this type, the authors suggest a very useful set of guidelines for designers of touch screen devices for children.

The next article is “Designing for Children’s Mobile Storytelling” by Sonia Franckel, Elizabeth Bonsignore and Allison Druin. It reports on a 9-month series of design workshops with children in which co-operative design methods developed at the University of Maryland were used to design mobile tools for story creation and collaboration. Of particular interest are the description of the design activities and the discussion as to how the designs from the children fed into the final product.

Taking a more theoretical turn, “Bits and Pieces: Potential Future Scenarios for Children’s Mobile Technology” by Michael Eisenbreg, Leah Buechley and Nwanua Elumeze reflects on the relevance of the von Neumann architecture in a mobile technology world and unpacks the input process and output of computing. Illustrated by examples from wearable and tangible computing, Michael, Leah and Nwanua challenge us as readers to think outside the box and present several interesting ideas for the community.

Christos Sintoris, Adrian Stoica, Ioanna Papadimitriou, Nikoleta Yiannoutsou, Vassilis Komis, and Nikolaos Avouris were quite literally challenged to think outside of the box in the final article of this collection. In their classical design paper, entitled “MuseumScrabble: Design of a Mobile Game for Children’s Interaction with a Digitally Augmented Cultural Space”, the authors document a case study focused on the creation of a mobile game for children. The authors describe in full detail the processes they went through, ending with the evaluation of the product.

Mobile phones have, undoubtedly, pervaded many aspects of our lives: no longer just a means of verbal and text-based communication, they increasingly support activities associated with web browsing, email writing, game playing, media consumption, and social networking, amongst others. Extending this further, mobile devices are increasingly being used not just for interaction with digital information, but also for interaction with the real world in which we live. The Mobile Interaction with the Real World (MIRW) workshop series has been running since 2006: its goal is to provide a platform for future research in this area. Volume 2, Issue 3 of the IJMHCI was a special issue on Mobile Interaction With the Real World. Guest edited by Enrico Rukzio, Niles Henze, and Xavier Righetti, this special issue represents a collection of five significantly extended versions of papers that were presented at the MIRW 2009 workshop. Readers are directed to IJMHCI Vol. 2, Issue 3 to read the detailed preface to this special issue in which the guest editors introduce and outline the challenges facing the field, as well as providing some background to the MIRW workshop series, and setting out future visions for the research domain.

The articles selected for the special issue cover the spectrum from novel interactions and applications for personal projectors, gesture-based mobile interaction with other mobile and fixed devices, a context-aware mobile shopping trolley, and a framework supporting the flexible interaction with remote displays.

“Projector Phones: A New Class of Interfaces for Augmented Reality” by Johannes Schöning, Markus Löchterfeld, Michael Rohs, and Antonio Krüger focuses on (a) the emerging trend of having both pico-projectors and cameras built into mobile devices and (b) their potential for new interaction techniques, applications and services, arguing that mobile projection is increasingly becoming ubiquitous and expanding the interaction space of mobile devices. Johannes and his colleagues discuss different spatial setups of camera and projection units and discuss advantages and disadvantages of each in terms of offering the potential for new interaction techniques. The authors also discuss two examples of applications for mobile camera-projector units that support augmentation of the real world: Map Torchlight is a mobile augmented reality application which tracks the orientation of a camera-projection unit in relationship to a paper map in order to project additional information onto the map; and LittleProjectedPlanet is based on the idea of using information from the real world (e.g., a track sketched on a whiteboard) as an integral part within a mobile augmented reality application (e.g., using the sketched track as part of a game).

Andrew Greaves and Enrico Rukzio introduce us to View and Share in their article entitled “View and Share: Exploring Co-Present Viewing and Sharing of Pictures Using Personal Projection”. Andrew and Enrico assert that co-present viewing and sharing of images on mobile devices is a popular but cumbersome activity. They present a framework for supporting the formation of ad hoc groups in which members of the group can engage in viewing and sharing of images. Their system supports image sharing with a single user, with multiple users, or with all users as well as allowing group members to borrow the projected display and allowing co-located users to privately view images (without the use of the projection). Andrew and Enrico report on a user study which illustrates the advantages and disadvantages of the interaction concepts and their implementation; their results provide interesting insights into users’ points of view regarding viewing and sharing based on personal projection, as well as social aspects and privacy issues associated with use of the system.

The third article – “Pervasive Computing in the Supermarket: Designing a Context-Aware Shopping Trolley” by Darren Black, Nils Jakob Clemmensen, and Mikael B. Skov – observes that, as stores integrate various technologies to support shoppers, shopping in the real world is becoming an increasingly interactive experience. Darren, Nils, and Mikael introduce CAST, a Context-Aware Shopping Trolley. Using context-awareness and acquisition of user attention, CAST helps shoppers locate and find products in a supermarket, helps them find optimal routes relative to, for example, a given shopping list, and provides them with appropriate product-related information. This article discusses the design of CAST, as well as its architecture and implementation. The authors also report on an empirical evaluation which highlighted that shoppers using CAST exhibited more uniform shopping behavior in terms of product sequence collection and ease of finding products than would be the case without CAST: as such, the authors conclude that CAST successfully supports the shopping experience.

The next article is by Jong-Woon Yoo, Woomin Hwang, Hyunchul Seok, Sung Kyu Park, Chulmin Kim, Woong Choi, and Kyu Ho Park, and is entitled “Cocktail: Exploiting Bartenders’ Gestures in Mobile Interaction”. In this article, the authors report on novel gesture-based interactions to support interaction with another mobile device or an interactive surface. Exploiting gestures employed by bartenders, the authors present a new gesture-based mobile interaction system: for example, users can “pour” multimedia data from one mobile device to another akin to a bartender pouring a drink into a glass. Similarly, a shaking gesture allows users to create new content by mixing existing data just as a bartender creates a cocktail by mixing a combination of ingredients. The authors report on an implementation of Cocktail and discuss user-based evaluation of the system.

Andreas Lorenz presents “A Conceptual Framework for Interoperability of Mobile User Interfaces with Ambient Computing Environments”. In this article, Andreas asserts that the use of mobile and handheld devices for interaction with remote services should be considered a promising interaction paradigm, allowing users to use their mobile phones as remote controls. He presents a framework which allows users to select the mobile device of their choice for interaction with a remote service. Andreas notes that, because of the heterogeneity of available devices and interaction styles, interoperability needs to be given particular attention by the developer: he describes the design of a general solution to enable mobile devices to have control over services at remote hosts. The applied approach enhances the idea of separating the user interface from the application logic, leading to the definition of virtual or logical input devices, physically separated from the controlled services.

The final issue of Volume 2 saw a milestone in the young life of the journal: that is, the publication of inaugural examples of comprehensive PhD thesis summary articles within the journal.

A lot of PhD work is published in a somewhat disjointed fashion as a series of papers, each covering distinct sub-components of the bigger PhD research picture. The result is that the collective whole of a PhD is often un- or under-appreciated. The IJMHCI invites researchers who have recently graduated with a PhD in a mobile HCI-related topic to submit a comprehensive overview article of their PhD research. Such articles provide recent PhD graduates with a unique opportunity to showcase to a broad reader base, in a comprehensive and centralized fashion, the ‘big picture’ of their research. In turn, it allows the mobile HCI research community to become aware of, to appreciate, and to take pride in the achievements of the newest members of our research family. These thesis summary articles serve to encourage us to embrace the fresh, new research themes and associated challenges emerging as a result of the exciting work of our latest graduates. Volume 2, issue 4 saw the first two PhD thesis summary articles published within the IJMHCI.

Preceding the PhD thesis summaries, an article entitled “Framing the Context of Use for Mobile HCI” by Satu Jumisko-Pyykkö and Teija Vainio explores the minefield that is context of use within the field of mobile HCI. Satu and Teija present an extremely useful, extensive, systematic review of HCI literature with respect to coverage of the issue of context of use. They reflect on the fact that context of use is still commonly (and, one might argue, inappropriately and unhelpfully) perceived to be a largely static, unchanging phenomenon – even for nomadic, mobile technologies. Based on their review, Satu and Teija contribute substantially to necessary clarification of the important concepts of mobile context of use. Furthermore, they have derived a descriptive model of context of use specifically for mobile HCI which, as presented in this article, highlights and enhances our understanding of the special characteristics of mobile contexts of use. Their model comprises five components of mobile context – physical, temporal, task, social, and technical and information contexts (each of which has identified subcomponents) – and, most importantly, identifies properties of these components at the levels of magnitude, dynamism, pattern, and typical combinations. Satu and Teija discuss the various facets of their model in detail within this article, and highlight the need to further understand and target the dynamic aspects of context – specifically, changes and transitions between and within contexts – for mobile HCI. They note that, although their model provides an overview of the characteristics of mobile contexts of use, it cannot be assumed that the different components and properties are equal – rather, importance of characteristics is dependent on the domain of study. Satu and Teija challenge us to look to how we might operationalise the components and properties of their model but suggest that their model can help both practitioners and academics to identify (and, perhaps, better understand) relevant contextual factors for mobile systems use.

In the first of the PhD summary articles, Eve Hoggan presents an overview of her PhD research in “Crossmodal Audio and Tactile Interaction with Mobile Touchscreens”. Eve asserts that the use of crossmodal auditory and tactile interaction can enhance the ability of users of mobile touchscreens to access data by non-visual means and can help to overcome problems that occur across various mobile contexts of use where one modality may be more or less appropriate than another. Eve promotes the notion of situational impairment brought about by the diversity and fluidity of context of use experienced by mobile device users – that is, users finding themselves temporarily unable (perhaps for safety reasons) to utilize a specific sense due to changes in context of use. In the “first formal investigations into crossmodal icons and the design of crossmodal audio/tactile feedback for mobile touchscreens”, Eve examines the manipulable parameters of non-speech audio and vibration-based feedback for crossmodal icon design, the performance achievable with multidimensional crossmodal icons designed on the basis of these parameters, the ease with which crossmodal icons can be designed into real-world applications and their impact on the usability of such applications, and the appropriateness of audio and tactile feedback across different contexts and situations. As a result of her extensive research, Eve presents the first set of guidelines to aid designers who wish to use crossmodal audio and tactile icons in touchscreen interfaces. Eve was supervised in her PhD research by Prof. Stephen Brewster, University of Glasgow, UK.

The final article, also a PhD thesis summary, is entitled “Visual Demand Evaluation Methods for In-Vehicle Interfaces”. It has been co-authored by recent PhD graduate Michael Pettitt and his research supervisor, Dr. Gary Burnett, University of Nottingham, UK. Michael’s research focuses on the development of a method for assessing the visual demand (or distraction) placed on drivers by in-vehicle information systems (IVIS). Michael compares two methods for measuring driver distraction: the occlusion technique and the keystroke level model (KLM). Recognising the limitations of the former as an economically viable evaluation method, Michael has developed and validated an extended keystroke level model as a means of predicting the visual demand of IVIS user interfaces. Michael asserts that one of the key benefits to his extended KLM is that it can be applied in the early design stages, thus making it employable prior to the development of robust prototypes. It goes without saying that the safety benefits that may be realized as a result of reliable means to evaluate the visual demand of IVIS user interfaces are immense, but Michael’s technique has potential for benefit across the mobile HCI community at large – for instance, anywhere the visual demand of a mobile UI design needs to be carefully considered.

I trust that you will agree that the articles comprising Volume 2 of the IJMHCI present excitingly diverse yet closely related concepts central to the field of mobile HCI. They have collectively delivered exciting research and innovation, practical guidance, and intellectual stimulation in terms of the research directions and challenges faced by the field of Mobile HCI. In so doing, I believe that this compendium, like its predecessor, serves as an essential publication for researchers, educators, students, and practitioners alike – especially those concerned with social and organizational issues associated with emerging mobile devices and the evaluation and impact of their use.

As I am sure most readers can appreciate, establishing a new journal literally from scratch takes more than a little hard work combined, most importantly, with patience. Over its second year of existence, as the journal has increasingly permeated the consciousness of researchers, educators, and practitioners in the field of mobile HCI, we have seen a consistent growth in the stream of submissions. Whilst excellent news for the IJMHCI, this places increasing demands on the journal’s amazing team of reviewers and associate editors. A very discerning team, they do a fantastic job of providing critical feedback on submissions and ensuring those articles that are ultimately accepted are of the highest standard. I would like to take this opportunity to thank all the members of the team who make the IJMHCI such a great publication – the invaluable contribution of the journal’s advisory boards and board of reviewers and, of course, the essential contribution made by the authors of published articles – which has not only led to another year of an exciting and vibrant journal, but has culminated in this comprehensive publication. Beyond the regular team supporting the IJMHCI are the people associated with the special issues of the journal – of which there were two in Volume 2 – and I would also like to extend my thanks to special issue guest editors and reviewers who help bring focused variety (an oxymoron, perhaps?) to the journal.

All that remains now is for me to welcome you to this book which draws together all the research achievements and challenges presented in the second volume of the IJHMCI.