The Trend of Published Literature on User Experience (UX) Evaluation: A Bibliometric Analysis

: The term user experience (UX) emerged in the early 1990’s. Thenceforth, UX has become a key term for researchers to focus on aspects that go beyond usability and particularly in the field of Human Computer Interaction (HCI). The aim of this study is to analyse the bibliometric aspect of UX evaluation literature from Scopus database whereby 644 papers were extracted. The study utilised publishing or perishing software to collect the data, while VOSviewer was used to visualise the data. Data analysis was also carried out using SPSS and Microsoft Excel. The publication of articles between 2018 and 2019 increased to 117 articles in 2019 and this is the highest publication to date. Most of the publications are from journals and conferences, mainly in English. Based on the analysis of the co-occurrence map of all keywords in the articles published, the keywords frequently used by the authors are user experience (416) and user experience evaluation (155). Most of the research related to UX evaluation was conducted in United States; and the researchers prefer multi-authored publications. The co-authorship map of the journal’s authors showed that V. Roto is one of the dominant co-authorships. Other than that, Arnold P. O. S. Vermeeren is also the most cited author of UX evaluation in Scopus database. This study presents the history of scientific literature in user experience evaluation and will provide guidance for future research.


Introduction
Over the last few years, user experience (UX) in Human Computer Interaction (HCI) has gained significance. UX has become an important component in product development process. UX in the HCI area refers to users' perception of the use of technical artefacts (Etinger & Orehovački, 2018). UX defines the subjective experiences, attitudes and emotional reactions of users when engaging with products and services, and cognitive behaviours of users as the fundamental elements of the experience (Filippi & Barattin, 2019). UX focuses on the relationship between product and user, and the feedback that contributes to the usage of contexts (Kiili, Ketamo, Koivisto, & Finn, 2014). Lallemand, Gronier & Koenig (2015) were among the first authors who used the term UX to explain all the dimensions of a person's experience with a system. UX is further clarified by the means and expectations of the user resulting from the use and/or expected use of a system, product or service which can build a preliminary personal evaluation of product quality (Hassenzahl, 2004). Having people's different response to any circumstance is the biggest challenge in UX. Besides this, depending on the context, the same person can have different feelings in the same situation. UX evaluation is becoming known by more and more people nowadays. UX evaluation, which is also known as the quality of user experience , is the process and result from the assessment of user on a product or system (Huang, Hong, & Xu, 2020). This is not only the product's final result of experience, but also a collection of ways of producing these experiences. Therefore, UX evaluation is also about identifying the results of user perception from the increasing complexity of product functions and interfaces, the emergence of new paradigms for interaction, the availability of emerging technologies and devices (Park, Han, Kim, Cho, & Park, 2013). The term that is related to UX or UX evaluation has been widely used among researchers around the world in order to investigate the perception of a user who uses a product or system before, during or after use. Thus, the aim of this study is to analyse scientific literature published by researchers who investigate UX using bibliometric analysis.

Bibliometric Analysis
Bibliometric analysis comprises the examination of data and text; which is one of the domains in the analysis. Bibliometric analysis is useful for identifying the patterns of large amounts of data from past literature and its significance is well acknowledged (Rusly, Ahmi, Talib, & Rosli, 2018). The concept of bibliometric analysis can also be interpreted as the academic activities and trends of research publications which are collected from online databases (Chen, Desai, Heyns, & Pietra, 2020) such as Web of Science and Scopus. Through this analysis approach, the results, including papers that are published in top journals and conferences, and research trends were identified. This approach can also explore various fields of research, most influential publications, top publishing location, and future research directions (Kamran, Khan, Nisar, Farooq, & Rehman, 2020). Therefore, this paper analyses research publications which are related to UX evaluation, which also includes the UX to give a meaningful overview of the UX areas for researchers to get ideas for their future research. The purpose of a bibliometric analysis is explained in this section, while the second section outlines the methodology adopted. The results of the related bibliometric that has been analyzed are then discussed in the subsequent section. In the last section, the results and recommendations for the areas of future studies are summarised.

Methods
The method used in this study is data collection from Scopus database as at 16 March 2020. The study collected all the data from Scopus database because it is the largest single description about the research and indexing the databases (Burnham, 2006). Scopus database is also one of the largest searchable databases for citation and the source for the researcher to do literature review in various areas such as economy (Oliveira, de Barros, de Carvalho Pereira, Gomes, & da Costa, 2018), health (Sweileh, 2018), and education (Ramirez & Devesa, 2019), to name a few. However, this study analyses the data and analytically produces the results from Scopus database which focus on the document and source types, the document year, language, author name, subject area, keywords, and country. The main purpose of this study is to focus on all documents that are related to user experience evaluation and user experience (UX). Document types are based on the title of the document, namely "Article Title". The query that was conducted in this study is: ((TITLE ("user experience evaluation") OR TITLE ("UX") AND TITLE-ABS-KEY ("user experience")). This query generates a total of 644 documents. These documents were used in this study for further analyses.

Results and Discussion
The data analysed in this study were elaborated by document type, source type, year of publication, language of document, subject area, country, authorships and citations. Tables, graphs, pie charts and other illustration forms were also used to illustrate these parts of the data. The results were presented as frequency and percentage until 16 March 2020. The results were discussed in the next section.

A. Document and Source Types
This study analysed all the data collected to identify their categorisation. The first part of the analysis was on document types and source type. Document types include conference papers, articles, review and book chapters. The source types are journals, conference proceedings, book series, books or trade publications (Rusly et al., 2018). Conference papers which appeared by type of document differed from those which appeared by type of source (Sweileh et al., 2017). An example of the term "document type" in which the conference papers are presented and are normally written as complete journal articles. Although some conference papers might be extracted by "document type", some other conference papers published in a conference or a book chapter might also be extracted as "source type". Table 1 shows that there are nine (9) types of documents which have been published based on UX evaluation or user experience. The types of documents are namely conference papers, articles, reviews, book chapters, books, notes, conference reviews, editorial and erratum. Table 1 also illustrates that most publications are conference papers (74.53 %), followed by articles (20.03 %), review (1.71 %) and book chapter (1.09 %). The other types of documents were less than 1 % such as books (0.78 %), notes (0.78 %), conference review (0.47 %), editorial (0.47 %) and erratum (0.16 %). Figure 1 shows the top five of document types. The findings showed that the authors prefer to publish their research papers in conference papers rather than in journals. This pattern shows document types such as conference review, editorial and erratum has not anymore become the favourite types among the academia to publish their research. Academics prefer to publish their research papers in the form of conference papers and articles because these are more familiar than the other document types. Moreover, this study discovered 5 source types that have been published which are related to UX evaluation or UX comprising conference proceedings, book series, journals, books and trade publications. Table 2 presents 5 source types that show the highest percentage: conference proceedings (52.02 %), followed by book series (22.98 %) and journals (21.89 %). The source type that also contribute such as book series (1.86 %) and trade publications (1.24 %). Based on Table 2, it is clear that conference paper is the priority source for researchers because it is the paper presented in a conference, and it is easier to understand. The frequency of conference proceedings was 335 compared to the series of book with a frequency of 148. The third source, with a frequency of 141, that is most referred to by the academia is the journal. Meanwhile, books and trade publications seem less popular by scholars as a source for reference.

B. Year of Publications
Another analysis is on the documents year of publications. This is shown in Table 3. The first article on UX evaluation was published in 1997 by Meedin & Perera (1997) entitled "Crowdsourcing Towards User Experience Evaluation: An Intelligent User Experience Questionnaire (IUEQ)". In this article, the author discussed UX design that contributes to customer satisfaction in any product, including smart phones. In 2010 and 2011, the increase in the corresponding publications increased slowly. Table 3 shows the dramatic increase in the publication related to UX evaluation. Figure 1 illustrates the projected number of publications in 2019, to be perhaps the highest since 2009. However, several publications have already been prepared and indexed in the Scopus database prior to 2019.  Table 3 below, the total number of publications by year was 644. From 1997 to 2008, only 9 publications were published. However, over the span of four years from 2009 to 2012, the number of publications slightly increased. Then from 2013 to 2017, publications by academics began to show an increase even though its fewer than 100 publications a year. Entering 2018 and 2019, the number of publications increased further, from 112 to 117 publications. Even though publications were lesser than 150 between these two years, publications were becoming more stable and academics were beginning to pay attention to the field of user experience (UX).  Table 4 shows that most of the documents were published in English (98.29 %). There were a few in Portuguese (0.62 %), followed by Spanish (0.31 %), Chinese (0.47 %) and Korean (0.16 %). Only one document was published in dual languages i.e. English and Spanish. Based on Table 4 also shows that the popular language for the papers studied is English. Analysis reveals that the frequency of publications in other languages such as Portuguese, Spanish, Chinese and Korean could not even reach 1% (based on 644 papers collected). This is because many of today's academic references are using English as the main language as it is more accepted and understood by researchers around the world.

D. Subject Area
This study also analysed the published documents based on their subject areas. Most of the studies on UX evaluation or UX were in the area of Computer Science, which holds 51.62 % of the total documents followed by Mathematics (13.37 %), Engineering (13.09 %), Social Science (8.45 %), and Decision Sciences (2.97 %). The other subject areas covered in UX evaluation or UX research are tabulated in Table 5. The analysis indicates the overall percentage for all journals in various fields is related to the concept of UX. There are five highest subject areas which applied the concept of UX. However, other areas such as business, management and accounting, arts and humanities, material science and medicine also applied UX concept in their studies whereby these subject areas carried a percentage between 1.11% to 2.04%. Figure 2 also presents the top five subject areas that substantially applied UX fields in their study. Other subject areas, which applied less than 1% UX concept or UX evaluation are Economics, Econometrics and Finance, Agricultural and Biological Sciences, Chemical Engineering, Earth and Planetary Sciences and a multidisciplinary subject.

E. Keywords Analysis
VOSviewer, the software instrument for creating and viewing bibliometric networks, has been utilised to map authors' keywords. Figure 3 shows a network visualisation of keywords used to address connections with other keywords in colour, square size, font size and the thickness of connecting lines. Keywords of the same colour, for example, have been commonly listed. In this study, for example, user experience, mobile applications, user experience design, and emotion carry a similar colour (green) suggesting that these keywords have a close relation and usually co-occur together. UX evaluation that co-occur together with augmented reality, virtual reality, game, prototype, questionnaire and device carries the same colour (purple). The keyword co-occurrence networks in Figure 5, also can display connections in the form of a network diagram between keywords. For example, the more relevant the keywords on the map, the closer the distance between the keywords. This close relationship forms a cluster and can accurately reflect a core topic of research ( Table 6 displays the top 20 keywords that are used in the UX evaluation and UX. Based on the number of occurrences after keyword data cleaning, it is clear that the keyword "user experience" is the most used (416). However there are also keywords that use both text and acroynm such as "User Experience (UX)" in the papers (155). Moreover, the keyword "Human Computer Interaction" also is one of the most used in the research papers. The keyword for "User Experience Evaluations" is at rank 7 (76). However, other keywords that are also popular among researchers are user interfaces (110), design (97), and human engineering (79). It is a surprise that usability keyword is at rank 8 because this keyword is one of the popularly used by scholars in UX research. Survey keyword is also among the top ten popular keywords for UX evaluation field. Based on the analysis, UX evaluation concept has been applied in many fields such as software design, user-centred design, software engineering, quality control and behavioural research. The finding shows that the field of UX evaluation has shown trends towards interdisciplinary research.

F. Geographical Distribution
The study found that there are 60 countries that contributed to the research publications. According to Figure  4, these countries partake in the distribution of published documents. The 5 countries with the absolute highest number of publications are the United States (US), United Kingdom (UK), Finland, Germany and Brazil. The US has led the academic frontier in UX research. It is therefore noteworthy to mention that the number of publications in the US is rising compared to other countries. Based on Figure 6, it is also evident that the US has close correlation with South Korea and India, Germany with Japan, Italy with Austria and Portugal. Moreover, Finland has correlations with Malaysia, Australia and South Africa. The study also analysed the total publications in different countries (based on the collected 644 papers), as shown in Table 7.  Table 7. The United States (US) is ranked first with a total of 147 documents followed by United Kingdom (UK) (58), Finland (46), Germany (43) and Brazil (40). There are other countries which also significantly contribute to the UX research even though the total number of publications is less than 20; namely Sweden, Austria, Australia, Canada, Netherlands, Denmark, Belgium and India. This analysis can be a reference for researchers worldwide to study the field of UX evaluation by country.

G. Number of Authors
The number of authors per document is indicated in Table 8. A total of 96 (14.91 %) records have been single-authored, while the others are multi-authored. There was no information on the authors' background on certain documents obtained from the database in Scopus. Although substantial work was performed to collect information on the authors, the effort failed. Based on Figure 5, the authors most frequently prefer to conduct research with partners: two researchers with 172 publications (26.71 %) and three researchers with 150 publications (23.29 %). It is worth to note that there is no multi-authored papers (beyond ten researchers) in UX areas. Table 8 exhibits the analysis using bibliometric approach. There seems to be no four-author collaboration in the publications; and this is with the frequency of 85, five co-authors with the frequency of 61, six co-authors with the frequency of 41 and seven co-authors with the frequency of 20. Multiple-author collaboration in publications has enabled publications to be well-written and thus improve the quality of publications. However, publications that are written by more than eight authors are with the frequency of less than 10. Most scholars prefer to write their research paper with only two or three authors. In the next section, co-authorship analysis of the author clusters will be discussed.

H. Co-Authorship Analysis
This research analyses co-authorship further with VOSviewer. The study is based on prominent writers with fractional counting approach that have more than one citation. The intensity of the relation between authors is indicated by colour, circle size, font size and thickness. Linked authors are generally grouped together, and shown by the same colour. The result indicates the clusters that are visible in output among the top seven clusters. This association within the clusters indicate that the authors have a correlation with each other in the field of UX evaluation. For example, the diagram below shows that Obrist, Tscheligi and Bernhaupt are collaborating closely or normally conduct research in the same research area (Figure 6), which is coloured in blue. The same analogy is shown in Law E. L. collaborating with Veemeeren, Hassenzahl or Hoonhout as a group of authors in the same research area and the linked is coloured in yellow. The analysis also showed that Roto is one of the most involved with co-authorship in the UX areas. Furthermore, Figure 9 displays the author's network virtualisation map that focuses on the countries in which the authors are associated with. This study only listed countries with more than three publications and more than one citation for the papers. Connected countries, as indicated by the same colour, are commonly coordinated together. The results indicate that the United States (US) plays a very prominent role in cooperating with other nations, based on the fractional counting method. The US has been working closely with United Kingdom (UK), Finland, Brazil and India, while UK with France, Canada and Belgium (yellow linked colour). Other than that, Italy appears to be coordinating with Denmark, China, Netherlands, Spain and Greece (green linked colour). Moreover, Germany appears to be coordinating with Indonesia, Norway, Japan, Austria, Switzerland and Portugal. Finland also shows coordination with Malaysia, Singapore, Australia and South Africa. This country visualization map also shows that co-authorship from these countries getting together in publication documents as indicated in Figure 7.

I. Citation Analysis
In order to obtain the citation metrics for the collected data, the analysis utilised Harzing's Publish or Perish program. Data that are collected from the Scopus database are imported into this program to create a citation metric as well as an absolute Google Scholar citation. As of 16 March 2020, the detail of citation metrics for the documents collected are listed in Table 9. The list includes the total number of citations, citation by paper and citation by author for each year. The year of citation in this analysis are from 1997 until March 2020 which involves 644 research papers from Scopus database. The citation per year is 127.30 whereby the total of 2928 citations are divided by 23 years of citation per year. While citation per paper is 4.55, in which 2928 total of citations are divided by 644 research papers analysed. The result for authors per paper is obtained by dividing the total of authors (2060) with the total of research papers (644). This analysis revealed that the research in UX evaluation fields is still new and require more research.   Based on the analysis done, the study can conclude that popular topics in UX evaluation research are related to product quality, product development, product design and product evaluation. The scope of research in the UX field has become broader, presenting a cross between different topics in the field of UX evaluation. The results are an interesting finding because they show that the most cited research papers are between 2008 and 2017, which are mostly conference papers. There are only three research papers that obtained at least more than 20 citations per year (Vermeeren et al., 2010;Hassenzahl, 2008;Hartson & Pyla, 2012) and cited by other scholars more than 200 citations. This finding shows the general trend towards greater interdisciplinary research that involves UX topics, and is starting to get extensive attention from the outside world.

Recommendation and Future Research
Among the suggestions for future researchers to be concerned with is to study the relevance of the authors that is analysed through bibliometric analysis. For example, the future researcher can provide detailed explanation on the top authors who appear in the co-authorship analysis in UX area, namely Roto, Veemeeren, Law E. L and Obrist. Moreover, future researchers can also elaborate in more detail about the relationship between authors who are linked together, represented by the same colour in the bibliometric analysis. Another suggestion is to provide further explanation on the area of articles studied by the authors such as Law E. L. with Veemeeren, Hassenzahl or Hoonhout (refer visualisation map of the network based on authors) for the same research area as a group of authors. Moreover, future research may analyse "Web of Science" database or make a comparison between "Web of Science" and Scopus databases in order to get more detailed information on the publications. Other than these suggestions, future researchers need to state the search query or coding that is used in their bibliometric analysis in detail. These recommendations may guide future researchers and offer possible opportunities for future studies.

Conclusion
This study reveals the areas that are mostly covered in the UX evaluation research are based on the keywords used by authors. Based on the analysis in this study, most of the document types are conference papers (480), followed by articles (129), reviews (11), book chapters (7) and books (5). The research on this topic was started in 1997 and has increased annually since then, as shown in the number of publications in 2019 (117) and 2018 (112). However, the analysis in 2020 only covers the date of search. The number of publications drastically increased in 2018 with a total of 112 publications, compared to 89 in 2017. The global distribution of the literature reveals that the United States has the most publications and has influences on the United Kingdom, Finland, Germany and other countries. In this study, bibliometric statistics of literature in UX evaluation revealed that publications were dominated by multi-authored publications. In addition, this study focused only on the UX evaluation subject, which also included UX based on the documents' titles. Literature which is relevant to UX evaluation but not included in the title specifically was omitted. It is imperative to note that no search results are completely correct. In this study, the analysis is based on Scopus and Google Scholar data. Despite these constraints, this study is one of the first to analyse the bibliometric information of the literature about UX evaluation and the analysis may be used by other researchers to get the overview of the UX area.