Tangible mixed reality for remote design review: a study understanding user perception and acceptance

Abstract

Background

The design review process is often expensive due to the need for face-to-face meetings between the involved parties. Distributed design collaboration is made possible by advances in networking techniques. A tangible Mixed Reality (MR)-based virtual design prototype was created as a distributed virtual environment (DVE) for the purpose of improving remote design review collaboration. This tangible MR system has been developed to a point that experimental evaluation is necessary in order to understand the strengths and weaknesses of various features of the system.

Methods

In this paper, we evaluated the tangible MR system against a commercial 3D distributed design review system in three aspects: the investigation of how users experienced virtual models in the tangible MR system as compared with the commercial system, the measurement of the users’ attitude towards the effectiveness of the tangible MR system, and the discoveries of usability issues involved in the tangible MR interface through usability studies.

Results

The findings from user feedback suggest that the tangible MR system may facilitate problem-solving and the quantity of work in a given amount of time and that virtual design displayed in the mixed scene was a useful aid in the design error detection task.

Conclusion

These findings are useful for the improvement of future generations of the MR system. Also the suggestions can be further generalized to become usability guidelines for the MR developers in other applications and domains.

Background

Distributed design collaboration is made possible by advances in networking techniques. For instance, there have been several well recognized interactive digital tabletop systems, either for collocated design groups or for networked collaborative design groups such as the TeleDesign interface of Shu and Flowers (1992) and the WebShamn (Tuikka and Salmela, 1999; Merrick et al., 2011; Singh et al., 2011). Interactive digital tabletop systems have been widely applied to many aspects of human life such as education (Khandelwal and Mazalek, 2007), office (Wigdor et al., 2007), military (DiGiovanni et al., 2005) entertainment (Cowell et al., 2004) and games (Nilsen and Looser, 2005). They offered novel and exciting approaches for humans to interact with various scenarios.

Computer-supported collaborative design review has been widely used Mavrikios et al., (2011). The collaborative conceptual design has been a major area of research work, mainly addressing several web- and agent based approaches to support collaboration during the early stages of product development, conceptual design tools and frameworks, conflict resolution, and team/project management for conceptual design (Wang et al., 2002). Extensive research work on collaborative computer-aided design (CAD) has also been reported, addressing issues, such as co-design systems and feature/assembly based representations, web-based visualization, 3D representations for web-based applications, and 3D streaming over networks (Khan et al., 2006). Several synchronous and asynchronous co-design systems, providing collaborative modeling functionality, have been developed. Most of them are based on the client–server model, while recently, some systems providing real-time online collaboration, based on a peer-to-peer network, have also been presented (Chen, 2001). The integration of different commercial client CAD systems into a co-design platform has been demonstrated in some cases (Huang et al., 2010). Virtual reality (VR) based systems for collaborative product modeling were also suggested in the past (Chryssolouris et al., 2009). Shared product visualization and collaborative design review has been another major area of research and development work (Dong and Kamat, 2013). Methods of sharing virtual product representations over the web and a number of CAD-integrated shared workspaces have been presented in the scientific literature for distributed design review (Hou et al., 2008; Hren and Jezernik, 2009). Shared VR-based environments have also been used to support interactive collaboration in product design review (Escorpizo et al., 2011; Liang et al., 2013). Most of the reported research activities focus on providing collaboration tools for specific phases of the product development process. Research works on internet-based product information sharing and product data management (PDM) related applications have also been widely documented (Hu et al., 2010). Apart from product design issues, several researchers have also worked on the development of methods and tools to support real-time collaboration for distributed activities related to manufacturing and product assembly (Zhen et al., 2010; Wu et al., 2012). The development of web-based manufacturing systems has also been extensively investigated (Lan, 2009).

Mixed Reality (MR) is a powerful user interface paradigm enhancing a user’s perception by incorporating the computer-generated information into the real world. Augmented Reality (AR), which is a sub-mode of Mixed Reality and sometimes interchangeable with MR in certain cases, can insert digital information into a predominantly real environment (Dunston and Wang, 2005). Verlinden et al. (2009) suggested the concept of augmented prototyping that projects the perspective images of the product on the physical object made by rapid prototyping techniques (Verlinden et al., 2003). Greenberg and Fitchett (2001) presented toolkits called Phigets that allow designers to explore a tangible user interface (TUI) for interactive product design. In addition to the toolkit, powerful tools including stereoscopic display systems, head mounted displays (HMD), data gloves and haptic devices have been introduced (Burdea and Coiffet, 2003) and combined to construct Virtual and Augmented Prototyping systems that provide realistic display of products in a simulated environment and offer various interaction and evaluation means (Wang et al., 2009). For instance, Bochenek et al. (2001) compared the performance of four different VR displays in a design review setting and mentioned that the best approach for design review activities could be a combined technology approach. As for AR adoptions for design review, Moeslund et al. (2003) and Sidharta et al. (2006) both employ optical see-through Head Mounted Displays to inspect designs, but little support is provided to the act of reviewing. Verlinden (2003) attempts to improve design reviews in the domain of Industrial Design Engineering by providing a specific prototyping and annotation device employing physical mockups. The IMPROVE project explicitly addresses design reviews for product design; its main focus lies with hardware and algorithms for photo-realistic rendering (Santos et al.,2007). The annotation facility is minimal.

Although various ways above have been proposed to support collocated Augmented Prototyping and design review of industrial and architectural products, more research is still needed in the following aspects. AR should be made available and accessible to geographically distributed people in product design evaluation. The prototyping environment should be available at low cost without strong restriction of its accessibility to remote users. Applications of AR have been predominantly focused on industrial and architecture product design, however, there is very little noted research on AR adoptions in the area of building services, which this paper is presenting. In this paper, we address these aspects by proposing a tangible Mixed Reality-based distributed virtual environment (DVE) — MRDVE — for distributed concurrent collaboration, which provides a platform which allows groups of people to share the same work and communications space over a network. There have been several noted research efforts that investigated the applications of distributed Mixed and Augmented Reality technologies for collaborative interior design applications (Ahlers et al., 1995), videoconferencing systems (Barakonyi et al., 2004), tutoring sessions in teaching multi-variate calculus (Orozco et al., 2006). Other examples include the collaborative augmented reality system Studierstube (Schmalstieg, 2002), SharedSpace (Billinghurst and Kato, 1999), and mediaBlocks (Ullmer et al., 1998). The development of MR/AR technologies for the architecture, design, and engineering industry should be guided by research aimed at adapting the features of the MR/AR systems to the different demands of the various situations in which it may be used. The user acceptance and usability issue are thus of great importance. There have been several attempts to evaluate the effectiveness of Virtual Reality technology in design and design collaboration. Usability testing has been widely implemented in Virtual Environments (Hix et al., 1999; Bowman et al., 1999; Gabbard et al., 1999; Hinckley et al., 1997). Although there has been much speculation about the potential of MR/AR, considerable work needs to be done in usability assessment of these mixed environments (Dias et al., 2003; Regenbrecht and Schubert, 2002; Tang et al., 2003). The common feature of these noted research efforts is the attempt to use well-designed systematic usability tests to create detailed knowledge about the usability aspects of the various interaction and information presentation techniques involved in remote MR/AR systems. Very few empirical studies have been done in the area of MR/AR, which creates a knowledge gap. Wang and Dunston (2008) implemented an experimental evaluation that was intentionally geared and implemented to investigate how users experienced virtual models in the MR system and to measure the users’ (1) experience, (2) satisfaction, and (3) attitude towards the effectiveness of viewing the 3D models through the MR system as compared with viewing those 3D models on paper.

The most state-of-the-art of commercial design review software provide environments for remote collaborative review. Since industrial construction practitioners are familiar with such remote design review tools, positive results from comparing the MR tool to one such commercial tool may persuade practitioners to move toward the adoption of MR/AR to change the way they work. Based on this motivation, experimentation on the tangible and distributed MR-based virtual space system is presented, a follow-up study to the one by Wang and Dunston (2008) mentioned above. The purpose of this experimentation is to gather and generate detailed knowledge, through well-designed empirical tests, about the usability aspects of the various interaction and information presentation techniques involved in the MR-based virtual space system from a practical perspective. Specifically, the experimentation assessed the effectiveness of the tangible MR system as compared with a representative commercially available remote collaboration system. Although the experimenters chose the mechanical design review as the context for evaluating the MR system, the results and learned lessons could possibly be lent to other similar types of design review activities.

Methods: System Prototyping and Experimental Evaluation

Prototyping the tangible mixed reality-based distributed virtual environments

Figure 1 depicts the generic system architecture of the MR-based virtual space system. This prototype was based on the previous version of an MR prototype introduced by Wang and Dunston (2008) for face-to-face design review collaboration but is a distributed version with extra remote functions and features. Commercial NetMeeting software is used for audio communication only. The hardware has been operated on a Pentium 4 PC with a 1.6 GHz processor (central server) and on different PCs and laptops (local clients) with head-mounted displays (HMDs). The components or devices consist of the central server, network, client computer, HMD, markers and arbitrary real objects. The central server is configured with a CAD application. The central sever can control what information is sent to each client. The network configuration connects server and clients, where an IP network with multicast support was realized on top of the network infrastructure. Client computers are configured with the same MR software, which may receive real-time online data packets for graphics objects. Multiple-marker recognition was used as the tracking technology where a high contrast tracking marker is essential. By manipulating the cube (real objects) to control the virtual tracking ball, users can naturally interact with the virtual object in the virtual graphic world, the users can tell the location of the real cube with reference to the virtual object. Therefore, by using the tracker cube, the user can track his/her hand’s position relative to the viewed virtual model by knowledge of the spatial relationship between the virtual model and the tracker ball. In the following sections, the tangible interface and virtual network computing components are elaborated because they are essential for implementing and understanding the user acceptance study presented in the latter part of the paper.

Figure 1

System architecture of the MR-based virtual space.

Tangibility

Tangible interfaces enable the integration of convenient arbitrary objects into the real collaboration environment for effective object manipulation. Tangible interaction techniques applied in the system are incorporated via tracking markers for Mixed Reality registration. The users can manipulate (rotate, tilt, move around) the virtual model design by manipulating the physical tracking marker underneath it. Video-based fiducial recognition was used as the tracking technology where a high contrast tracking marker is essential. The computer performs image processing on the video image captured from the camera to find the specially marked tracking marker(s). Once an appropriate marker is recognized, a virtual model is then registered to the position of the marker. The resultant composite image reveals the insertion of the 3D virtual model into the user’s real world view. Whenever the spatial relationship between the camera and the tracking marker is changed, the virtual model moves in concert with the marker. The other key tangible interaction technique applied is a tracker cube used for interpreting the spatial relationship of virtual and real entities. The tracker cube is actually a specially marked, small-size real cube with a virtual ball overlaid on it (see Figure 2).

Figure 2

Illustration of the virtual tracking ball for positioning through manipulation of a real cube (Wang and Dunston 2008 ).

View windows

The system design is based on a distributed shared Mixed Reality scene window. When communicating the model with others over the network and transferring attention back and forth between discussion with other people and observation of his/her own 3D models, the user must periodically share the perspective and focus with the other collaborators in order to ensure consensus in their decisions. Such consensus can enable collaborator A to keep track of the series of actions taken by collaborator B from the latter’s viewpoint. For example, collaborator B asks A to navigate to a target by passing three walls in front and then turning 60 degrees left and then passing another five walls forward in a virtual environment. Collaborator A might be confused by such verbal instructions and might fail to reach the target. An effective collaboration is thus difficult to accomplish. Effective perspective sharing through certain groupware software can reduce such cognitive load that is required otherwise.

Experiment setup

As stated above in Section 1, the entire experimentation was implemented using the procedure/method that was set by the authors in a study comparing face-to-face collaboration facilitated by paper and MR media (Wang and Dunston, 2008). This procedure was used in that instance to measure the users’ perspective on a tabletop Mixed Reality collocated application. The remaining discussion details the evaluation of the MRDVEs using the same method. The experiment aimed to collect data for investigating the subjects’ experience with the MR system, gauge the subjects’ attitude towards the MR system, and discover usability issues. Three questionnaires were administered to individual participants in an experiment where pairs of subjects worked collaboratively using both the MR-based virtual space system and a leading commercial design review application to detect design errors: the first questionnaire was devised to measure the users’ attitude towards the effectiveness of the MR system; the second questionnaire investigated how users experienced virtual models in the MR system as compared with the commercial system; the third questionnaire aimed to reveal usability issues involved in the MR system interface through a usability study.

Comparison benchmark: vommercial3D design review tool

The commercial design review application chosen for comparison, NavisWorks Roamer, is sophisticated commercial 3D design review software which enables geographically distributed users to review 3D designs (see Figure 3). For interaction with models in NavisWorks Roamer, there are nine navigation modes to control how you move around the main navigation view six camera-centric modes (walk, look around, zoom, zoom box, orbit, fly) and three model-centric modes (pan, examine, turnable). In a camera-centric mode, the camera moves within the scene, whereas in a model-centric mode, the model moves inside the scene. NetMeeting software is incorporated into NavisWorks Roamer as a built-in functionality. It was the experimenter’s responsibility to install the marker in its precise position and calibrate the MR system. The marker’s final relevant position was not subject to be altered afterwards. For example, in order to guarantee that the camera did not lose the tracking image, the marker was set still in the space. The human subjects only needed to familiarize themselves with the MR interface and how to interact with the MR information.

Figure 3

Screenshot of NavisWorks Roamer with redlining and interaction functions demonstrated.

For collaborative design review, NavisWorks Roamer essentially supports asynchronous collaboration, where one collaborator finds the suspicious error, redlines it, saves the current viewpoint, and then sends the information to the other collaborator. The other collaborator then receives the file and loads the saved viewpoint for review. However the MR interface support synchronous collaboration and thus allows real-time view slaving and sharing. Based on the collected data, the time spent on sending views in NavisWorks was quite short, averaging less than half a minute, which almost constitutes synchronous collaboration. This situation is similar to sending and receiving emails between two persons who are waiting at both ends for each exchange. Furthermore, during the experiment, the experimenter intentionally excluded recording the time that was spent on file transfer in order to make its comparison with MR fundamentally fair. A stop watch was used to capture the task duration without file transfer. When the subjects filled in the questionnaires, the experimenter alerted them to not consider the file transfer time as a drawback/effect when rating/evaluating the NavisWorks performance.

Hypotheses

As noted above, post-experience questionnaires were used to collect data for investigating the subjects’ experience with the MR system, gauge the subjects’ attitude towards the MR system, and discover usability issues. Our prior experience in testing applications of MR technology has demonstrated that user issues of visual quality, maintaining spatial awareness, and functional utility (i.e., comfort and ease of use) are important to potential users embracing such a system (Wang and Dunston, 2006; Wang and Dunston, 2008). Therefore, the first questionnaire implemented was designed to evaluate the subjects’ experiences of using the tangible MR-based virtual space system and NavisWorks Roamer in terms of the following five hypotheses:

•  Hypothesis 1 asserts the following: When compared to NavisWorks Roamer, the MR-based virtual space system will involve significantly better quality of visual presentation;

•  Hypothesis 2 asserts the following: When compared to NavisWorks Roamer, the MR-based virtual space system will induce significantly higher physical comfort;

•  Hypothesis 3 asserts the following: When compared to NavisWorks Roamer, the MR-based virtual space system will involve a significantly more natural and intuitive interaction with the model;

•  Hypothesis 4 asserts the following: When compared to NavisWorks Roamer, the MR-based virtual space system will involve a significantly higher level of immersion;

•  Hypothesis 5 asserts the following: When compared to NavisWorks Roamer, the MR-based virtual space system will give users significantly more ability to maintain sense of location and orientation.

Subjects, tasks, materials and method

Sixteen (12 men, 4 women) graduate student participants were recruited for the study (8 groups with 2 in each group). The participants’ ages ranged from 25 to 33. Important to the backgrounds of those graduate students was their lack of any 3D computer-aided drawing and animation experience, which practically eliminated experience-based bias. While selecting the subjects, a screening was conducted before the experiment in the format of a simple oral interview. All participants were pre-screened to ensure they had no sophisticated experience with CAD, VR, and MR applications. The modules for design review in the experiments were all taken from real construction industry designs and thus a proper level of complexity in the modules was ensured. These precautions can make the experience factor insignificant to bias the results. Furthermore, none of the participants had any known impairment in spatial cognition and reasoning, any brain damage, or eye (vision) problems that were not corrected by standard eyeglass prescription. It is important to note that there was no cross-over between participants in this experiment and the earlier experiment by Wang and Dunston (2008) comparing the MR approach to the paper-based method. Thus learning curve effects and related biases were avoided as well.

Industry participants in design reviews typically want to view and browse the design independently. Therefore, the experimental task was to review and inspect the design model independently and then figure out the design errors. The error detection task was chosen because it emphasizes design model comprehension and opportunities for collaboration and it involves a high demand of mental interpretation and maintaining a viable spatial mental model. Real 3D CAD models, obtained from an industry partner, were used as designs to be reviewed using the two methods. Small-scale cluttered ductwork models were used for collaboration between two individuals in separate rooms but connected over the network. Before their collaborations, each team of participants was briefed consistently on the types of errors that might be found. Two ductwork model designs were confoundedly used for the two treatments (MR and NavisWorks Roamer) respectively (see Figures 4 and 5). Both designs had a similar spatial layout but different locations of the errors. Three questionnaires were designed to be completed by the subjects with the purpose of investigating the users’ experience with the MR system, measuring users’ attitude towards the MR system, and implementing usability studies.

Figure 4

Design Layout of AutoCAD Design I for the two treatments.

Figure 5

Design layout of AutoCAD design II for the two treatments.

Treatments

The Treatment 1 was to perform the review using commercial NavisWorks Roamer where the pair of subjects could interact with the 3D ductwork design in a desktop window environment (see Figure 6a) in two different rooms. The Treatment 2 was to use the MR system (user can move his/her marker in hand and view the model from different angles via HMD) where the subjects in two different rooms were able to see their own virtual ductwork model floating above the corresponding real tracking marker via their HMD (see Figure 6b). Subjects can change viewpoints by either moving around or manipulating the real tracking marker underneath the virtual ductwork model.

Figure 6

Screen Shots of two Visualized Collaboration Treatments. (a) NavisWorks View and (b) MR View.

Experiment process

Each experiment session lasted three hours, including training session, actual experiment session, and post-experiment questionnaire session.

•  Training session: The subjects were instructed to use two treatments/methods (MR and NavisWorks Roamer) to review a design. They were assigned enough time to practice how to use the different platforms.

•  Actual experiment session: In the real experiment, each team consisted of two subjects (collaborators): one in office A that “owns” Part I of the design and one in office B that “owns” Part II. Each collaborator reviewed the design of the combined models. Each group performed a trial with each method (treatment) to check the two respective sets of the model in a specified sequence. During this process, the subjects checked the design errors (see Figure 7 for an example).

•  Post-experiment questionnaire session: At the end of the session, each user or group was asked for their comments and observations on the system, as well as presented a set of questionnaires on the comparison of the MR system and NavisWorks Roamer regarding various features

Figure 7

An example of design error pattern: interference between two rectangular ducts highlighted in red.

The participants’ performances were measured by subjective matrices, such as questionnaires, as well as objective matrices, such as task achievement and duration. It was generally observed that the subjects using the MR system completed the task much faster than the subjects using NavisWorks Roamer. It is believed that the perception and spatial cognition advantages of the MR system resulted in an offloading of some mental processing to the MR system and thus enabled the subjects to complete the task in less time. Further details of the time comparison are described elsewhere by Wang (2005). Furthermore, special useability questionnaires and associated data collection strategies like interviews and observations were developed in order to assess the participatory process and certain features of the MR space. Finally, a statistical model was also developed to arrange experimental sessions and collect data. A statistical analysis parameter (p-value) was used to test inter-factor correlations for reliable results.

Development of questionnaires

The first questionnaire was developed to evaluate the subjects’ individual experiences of the virtual model in the MR system as compared with how they experienced the design in the NavisWorks Roamer environment. The questionnaire ratings were on a five-point scale, with a value of 5 representing excellent and 1 representing poor. The questions and a summary of the results are presented in Table 1, including average ratings and standard deviations for each of the methods’ features. Aspects covered include quality of visual presentation, physical comfort, naturalness and intuitiveness of interaction with the model, level of immersion, ease of navigation, ability to maintain sense of location and orientation, overall suitability for making decisions and performing tasks on design models.

Table 1 Evaluation of the subjects’ experiences of using the MR system and the NavisWorks Roamer, rated from 1 (poor) through 5 (excellent)

The second questionnaire was developed to evaluate the subjects’ experiences of the virtual model in the MR-based virtual space system for conducting collaborative work compared with their experiences in the NavisWorks Roamer environment. In order to minimize the questionnaire’s bias and/or order effects on the results, half of the subjects evaluated the MR system relative to the NavisWorks Roamer, as in Questionnaire I, while the other half evaluated the NavisWorks Roamer relative to the MR system, as in Questionnaire II (see Table 2). The five-scale rating is from “totally agree” to “neutral” to “totally disagree”.

Table 2 Statements for soliciting measuring the subjects’ opinions on the value of the MR system for collaborative work

The third questionnaire aimed to investigate further some of the questions not fully addressed by the previous questionnaires. The questionnaire was used to explore the usability issues involved in the current version of the MR system and identify the need for any remedies. The questions in the questionnaire were answered using a five-step scale where 1 _represented very little and 5 _represented very much. Table 3 illustrates the mean ratings and standard deviations for the 15 usability questions. The second column lists the usability issues considered; the third gives the mean rating value and standard deviation for each usability issue from the opinions of the subjects.

Table 3 Means (S.D.s) for usability questions

In Table 3, Questions 1–11 were designed to measure the influence of the interface of the MR system on the effectiveness of the subjects’ performances. Questions 12–15 were more related to the comfort issues involved in using the MR system. Comfort was measured in the areas of arm strain, neck strain, dizziness, and nausea.

Results and Discussion

Questionnaires

A one-way analysis of variance (ANOVA) was performed on the individual ratings for both the MR system and NavisWorks Roamer. The results of mean ratings, standard deviation, F-values, and p-values for comparisons are illustrated in the Table 1.

Initially, it is noted that the subjects rated the quality of visual presentation of the model viewed from the MR system (Mean, 3.6; S.D., 0.74), which is marginally higher than the NavisWorks Roamer (Mean, 3.1; S.D., 0.99). The difference is not significant with p-value (0.1543) > 0.05 and F (1, 14) = 2.14, which does not support Hypothesis 1. This is different from the results obtained from the previous experiment by Wang and Dunston (2008) where an MR system was proved to provide more realistic and vivid virtual models than a corresponding 3D color drawing on paper. The virtual model in the MR-based virtual space system possessed a similar level of fidelity as the 3D model in NavisWorks Roamer, however, the current quality of the virtual model in the MR system is somewhat limited (bottlenecked) by the resolution of the head-mounted display. The current virtual models in use were converted from original 3D AutoCAD models through the PolyTrans Converter, where certain rendering information is lost. A more sophisticated converter might preserve more rendering data, therefore obtaining a higher level of fidelity in the MR virtual model. Some subjects commented that more shading and shadow rendering might further improve the visual quality and thus enhance the user’s spatial cognition.

The subjects felt more physical comfort using NavisWorks Roamer (Mean, 3.27; S.D., 1.22) than the MR system (Mean, 2.73; S.D. 0.83). However, the significance was not attested with only a p-value (0.1602) > 0.05. This does not support Hypothesis 2. The users’ neck and back strain (approximately 10–20 minutes of use) in the MR system was induced by the long-time uncomfortable poses for inspecting details of the model and the physical weight of the HMD.

Interaction with the virtual model via the nine functions in the NavisWorks Roamer seemed to be as natural and intuitive as in the MR system according to the ratings (NavisWorks: Mean, 3.26; S.D., 1.03; MR: Mean, 3.87; S.D., 0.64). The difference is insignificant according to the p-value > 0.05, and therefore Hypothesis 3 is not supported. This similarity can be explained as the trade-off between the unnaturalness of MR due to the demand of more physical effort and the intuitiveness of interaction with the virtual model via the virtual tracking ball. The subjects assessed the level of immersion for the MR system at an average rating of 4.37 (S.D., 0.81) compared with 2.73 (S.D., 0.88) for the NavisWorks Roamer with a significant p-value indication, which strongly implies that the MR system provides the users a greater sense of being present with the model. Thus Hypothesis 4 is well supported.

Subjects did not identify much difference in the ease of navigation between these two treatments, rating both of the tools as providing a mediate ease of navigation (MR: Mean, 3.87; S.D., 0.99; NavisWorks Roamer: Mean, 3.2; S.D., 1.15). However, for inspecting the details of the design model, the MR system enabled the users to move more easily to a more advantageous viewpoint (closer and directly in front of the target) to complete the error verification more quickly. This advantage was observed by the experimenter in a large number of trials. The subjects had learned the layout of the 3D design space and expressed more confidence in the error detection task without further effort to interpret the occlusion and depth cues among the objects because they were reviewing the design model in a 3D manner.

Hypothesis 5 is well supported because subjects using the MR system had higher ability to maintain sense of location and orientation as they navigated the model than when using NavisWorks Roamer (MR: Mean, 4.1; S.D., 0.7; NavisWorks Roamer: Mean, 3.2; S.D., 1.16). The difference is regarded as significant with a p-value (0.0305) < 0.05. Many subjects commented that there was easy loss of spatial orientation in using NavisWorks Roamer. NavisWorks Roamer provides the subjects with the bird’s-eye view (exo-centric) and close-view (ego-centric) for review. As the subjects tried to switch between these two views through the viewing functionalities (e.g., zoom out from a small object to the complete view of the design model), they had to mentally transform the view in their mind to accommodate the change of these two reference frames. Therefore it was often observed and reported that the subjects became lost or disorientated, especially after using the interactive viewing techniques to switch between these two views. This disorientation and re-accommodation was an additional mental burden to the cognitive attention switching. Some subjects using the NavisWorks Roamer reported that they felt frustrated because it was hard to even know the previous viewpoint of the design and they had to spent additional time to learn the model again from the new perspective. In contrast, fewer cases were reported from the users of the MR system since MR provided a smooth and natural way to transit back and forth. The subjects using the MR system could offload this centricity transformation task to the intuitive viewing mechanism of the MR system. When the subject leaned back allowing the camera mounted on the HMD to capture the complete design, a bird’s-eye view (exo-centric) was created. For inspecting the details of design model, the MR system enabled the users to move their viewpoint to a more advantageous position (closer to the target, with the target directly in front of them) to complete the task more quickly. When the subjects approached a target closely enough, an ego-centric view was created. The extra time spent in using the NavisWorks Roamer to accommodate the switch of reference frames was certainly mostly cognitive overhead. Therefore, the frequent loss of orientation could be interpreted as the inefficiency margin of the NavisWorks Roamer against the MR system in the subjects’ spatial orientation activity.

The final point of comparison asked for a general impression of the two methods. Subjects believed that the MR system was more suitable for making decisions and performing tasks on design models than the NavisWorksRoamer system.

Results of questionnaire 2

For the convenience of interpreting the data from the two questionnaires in Table 2, it was rationalized that the respondents who ‘totally agree’ with a statement of Questionnaire I were regarded to ‘totally disagree’ with the corresponding statement of Questionnaire II. Therefore, the percentages for these responses were added together. A similar summation of results was applied to the other four levels of the response scale. The data from the two questionnaires was thus tallied with respect to attitudes regarding the MR system and is visually presented in Figure 8.

Figure 8

Plot of Responses to Attitude Questionnaire.

As indicated in the Figure 8, about 69% of the respondents felt that 3D interactivity in the MR system aided design comprehension in total or partial agreement with the statement. Question 2 asked if the MR system overall better facilitates design collaboration tasks than the NavisWorks Roamer, with which about 69% of respondents agreed. Most of the respondents (63%) believed that the MR system better facilitated communication and creativity and a total of 65% of the respondents totally or partially agreed with the statement that the MR system better facilitated problem-solving. The increase in the overall quality of output from the collaboration is marginally positive (57%), however, the belief that the MR system better facilitated the quantity of work in a given amount of time and increased the quality of the user’s contribution to the project is more marked (70% and 70% respectively). It is surprising that these impressions are not stronger in light of the clear time benefits that were observed. The belief that there has been an increase in self-satisfaction from the collaboration as a result of using the MR system is well supported by 63% of the users. Only one-third of respondents, however, believed that the MR system increased understanding between their collaborator and them. One-third held a neutral attitude and one-third disagreed with the statement. A summary of the results convey the conclusion that the MR system both aids individual performance and also supports the collaboration, but there are implications that improvements are yet desirable.

The MR system was rated to offer the advantage of improved collaboration as expected. By sharing the same viewpoint continuously and synchronously between two remote subjects via VNC platform, one subject could maintain the same changing of viewpoints and frame of reference along with the other subject who led the discussion and was in charge of navigating in the MR environment. On the contrary, the subject using the NavisWorks Roamer had to iteratively go through the process of finding the suspicious error, redlining it, saving the current viewpoint, and then sending the information to the other subject. The other subject then received the file and loaded the saved viewpoint for review. Therefore, the receiving subject had to execute additional mental transformation to adapt to the static viewpoint saved by the sending subject for creation of a discussion context. The subjects’ task performance using the MR system was therefore improved by reducing the frequency of such electronic exchanges of the static red-lining comments and saved viewpoint between subjects.

Results of questionnaire 3

This section explored some of the usability problems identified from Table 3 and possible redesign recommendations as given for the future generations of MR-based virtual space systems. The following discussion and interpretations are based on the selected questions from Table 3.

Questions 1, 2, 4, 6, 7, 8, 9, 14, and 15 deal with the usability of the HMD. From the ratings in the first question, the respondents reported that the MR visual display did create difficulties for performing (Mean, 4.31; S.D., 0.96). The difficulty mainly resulted from having to wear the cumbersome and uncomfortable HMD. The fact that one hand of the user needed to hold the HMD to maintain a stable view of the mixed scene, increased the arm muscle fatigue. Also, the screen in the HMD was designed to make the user feel that the screen was actually placed several feet away from the eyes. The faraway and low resolution display resulted in a low clarity of the virtual model compared with a standard computer monitor. The fidelity of the HMD is totally dependent on the advance of technology in the industry. More advanced displays might be a promising option for exploration and development. HMDs are well-suited for applications (e.g., architectural and interior design) where complete visual immersion or absence of distractions is required. From the relatively high rating in question 4 (Mean, 3.75; S.D., 1.13), it seemed that the users did not feel much distracted by outside activities (distractions), which implied that the HMD was useful in focusing the users’ minds on the task at hand. This result is consistent with the subjects’ assessment of the level of immersion for the MR system at an average rating of 4.37 in Table 3. For question 6, the rating for appropriateness of the FOV for supporting this activity was relatively low (Mean, 2.63; S.D., 1.20). The broader the FOV, the better sense the user should have for the environment and communication with other users. This sense is much reliant on the HMD and video camera as the subjects’ FOV is constrained to the governing FOV of the display or the camera. The limited field-of-view of the video camera currently used, constrained the user’s view to some extent. The visual output had moderate stability of the image as far as allowing the subjects to move with no perceivable distortions in visual images (Mean, 2.81; S.D., 1.17) as indicated by Question 7. The stability of the vision tracker (fiducial recognition) is still insufficient for ensuring stable Augmented Reality applications. Thus improved or alternative methods are required for maintaining a proper registration. The further substantiation of the tracked six-degree-of-freedom data by redundant tracking methods can be a feasible solution. Hybrid technology tracking systems should be examined for applicability and performance regarding stability. In response to Question 8, respondents thought the MR system had a relatively acceptable degree of response delay with no perceivable distortions in visual images (Mean, 3.31; S.D., 1.14). Thus, the system lag was tolerable and did not affect the perception of the visual image by users and therefore did not hinder their performance. The subjects experienced a rather convincing scene of models appearing as if in the real world (Mean, 3.88; S.D., 0.88) as reported for Question 9. The virtual model looked convincingly to be floating in the air of the real environment, which was due to the stereoscopic feature of the MR display with two display panels creating high object presence effects. The relatively high rating implied that the combination of virtual model and real world reached seamlessness to some extent. It is interesting to note regarding Question 14 that the rating (Mean, 3.5; S.D., 1.09) implies that some subjects might have experienced light eye fatigue. The subjects normally wore the HMD for only 20–30 minutes for the entire experiment. If a user has to wear it for a longer time, eye fatigue might emerge more prominently as a negative. On the contrary, regarding the last question, the subjects experienced very little nausea during their interaction with the MR system (Mean, 1.75; S.D., 1.0). These issues need to be further examined and addressed if long-duration tasks are to be performed. It was also revealed from some additional short-answer questions in the questionnaire that about 87.3% of responders believed that they would not be resistant to using MR or similar MR systems in the future. About 81.3% believed that they would embrace the opportunity to use the MR system again in the future. Almost none of the users experienced “blurred vision”, “dizziness”, “nausea”, “difficulty focusing”, or “loss of vertical orientation” after exposure to the MR system.

Questions 3 and 13 deal with the naturalness of interaction. In response to the question whether the subjects had a natural perspective that gave them a compelling sense of their hand-motion while manipulating the tracking marker (Question 3), the rating (Mean, 3.38; S.D., 0.88) indicates a mild agreement. From Question 13, the subjects revealed a belief that the tracking marker was lightweight, portable, non-encumbering, and comfortable thereby avoiding issues of limiting their mobility and fatigue (Mean, 3.94; S.D., 0.85). These results are consistent with the rating (Mean, 3.87; S.D., 0.64) from Item 3 in Table 1.

Questions 5, 10, 11, and 12 deal with the issues regarding the overall control of the MR system. The surrounding real background seemed to marginally help the subjects’ spatial comprehension of the model (Mean, 3.56; S.D., 1.15) as evidenced by responses to Question 5, an expected result. The real background could have served as a stationary landmark reference for orientation of the virtual model, especially while users were turning the tracking marker. Question 12 responses indicate almost all the subjects had a fairly uncomfortable experience with long-term use of the MR system, with a very low mean rating of 2.06 (S.D., 1.28). This result is in accordance with the high physical discomfort rating in the first questionnaire discussed earlier. The bulky HMD is the most significant usability issue of the MR system. Because the design review collaboration task might require long-duration, intensive work, the MR system would require moderately high levels of ease of use and user comfort. It was found that using the MR system produced neck strain in some users. The tracking and interaction techniques in future generations of the MR system must be easy, comfortable, and intuitive to the user, so that the focus can be on the task and not on the system itself.

Conclusion

Based on the experimental evaluation of the MR-based virtual space system, several conclusions were noted. It was observed that the subjects using the MR system completed the task much faster than the subjects using the NavisWorks Roamer commercial visualization software. It appears the increased 3D visual perception, spatial cognition, and spatial layout of the design offloaded the required mental processing to the MR system and thus enabled the subjects to complete the task in a shorter time. Subjects felt more physical comfort using NavisWorks Roamer than the MR system. The subjects rated the level of immersion for the MR system, on average, much higher than for the NavisWorks Roamer application, which implies that MR definitely provides the users a sense of being present or isolated with the model, more than with the static model in NavisWorks Roamer. The subjects believed that the MR system was more suitable for making decisions and performing tasks on design models than the NavisWorks Roamer. MR seemed to better facilitate communication, creativity and problem-solving. The increase in the overall quality of output from the collaboration is marginally positive, however, the belief that the MR system better facilitated the quantity of work in a given amount of time and increased the quality of the user’s contribution to the project is more marked. Perhaps more importantly, there is a belief that there has been an increase in self-satisfaction from the collaboration as a result of using MR.

The usability studies implemented with the third questionnaire revealed usability issues and meanwhile helped to produce possible remedies. These suggestions are useful for the improvement of future generations of the MR system and summarize below Also these suggestions can be further generalized to become usability guidelines for the MR developers in other applications and domains.

•  Participants suggested a zoom feature to facilitate inspection of some tiny objects inside dense model structures.

•  Participants expected a quicker response in VNC for the real-time collaboration. There was still some latency (lag) in synchronizing the MR scenes between the two collaborators because the VNC conveyed the video stream which occupied too much of the available computing resource. The performance was heavily dependent on the computing resource and the network bandwidth.

•  Participants suggested a better user-friendly display. The current HMD was not comfortable enough for long term use. The HMD kept falling and needed one hand to hold it to a correct position, which created extra hand occupation and distracted the users from focusing on the work at hand.

•  Participants requested a camera with an automatic focusing function. If the user pulled his or her head back farther, the camera failed to recognize the tracking marker even though it was actually in the FOV of the user.

•  Participants suggested adding shadows and shading into the virtual design to help the users obtain more depth cues for better apprehension of the spatial relationships.

Even though these experimental results are preliminary, they proved useful in deriving guidelines and further developments and experimentation. Learning from the preliminary results of the usability study, the authors now have to refine the MR interface to tap into more potentials (Wang et al., 2012). One important factor not addressed specifically in the experiment was the subject’s level of expertise. The testing of expert users could produce quite different results. There are very few potential users of most MR applications who could be considered experts, so the current results are useful, but an understanding of how performance is affected by greater experience with conventional tools would have an added value. Future work will hopefully lead to the implementation of the MR system on real construction projects. The real improvements in performance and productivity with MR can be then measured and quantified with site assembly activities in a real project context. The transfer of the animated MR system from laboratory-based applications to real construction applications has higher potential for system flexibility and tracking, e.g. to enable assembly in a limited way on construction sites. Using portable MR devices such as wireless HMD or cameras will enable more stable tracking (images will not be lost when occluding the path between tracking targets and camera). Where the tracking targets (markers) are not able to be posted, markerless tracking techniques such as tracking using the salient geometric features of real spatial objects might be adopted. Since tracking is one of the most essential elements in determining whether the MR system is effective or not in real projects, future work should focus on the integration of the current tracking technologies and develop more robust tracking methods.