Card Sorts by Multi-Ethnic Users

cardsort-illustration Most everyone agrees that web site design should be tailored to the specific needs, moraes, and habits of groups based on culture. However, some situations involve the interaction of groups from many different cultures at once. Examples might include social media users networked with people worldwide; multi-national business sites; and educational support systems combining US and international students.

A recent user study that we conducted with multiple participants (24 US citizens and 24 non-US citizens) located in the United States provides an inventory of the specific preferred elements (icons, labels, menus, language elements) that constitute a shared web interface as well as how users actually perform tasks on a shared site (Sapienza 2008). Additional research was conducted at the time about how the users cognitively imagine and map out websites through a card sort process but excluded from the final article. A preliminary account of the card sort part of the research is provided here.

What Is Card Sort?

Card sorting is one common method by which usability researchers and practitioners obtain some conception about how users organize information. Users are presented with a set of cards labeled with specific topics that appear (or might appear) on a website and are asked to group them into categories that makes sense to them. A closed card sort provides overarching topic categories for each group in advance, while an open card sort allows users to decide for themselves what the categories should be called. The results of a card sort can be statistically compared and calculated through cluster analysis, multi-dimensional scaling, and other methods. Many usability software applications now provide powerful analytical tools for card sort implementation and analysis.


Users in each group of 24 were given a set of 17 topic cards extrapolated from a university computing technology website. Users were asked to self-select categories or groupings for the cards and supply a category title for each group of their choosing. However, some users used an existing card from the deck as a category or made slight changes to existing cards used as category markers rather than create new categories. In these respects, while users were free to perform an open card sort, some ended up doing something closer to a closed sort. During the analysis phase, category group labels were streamlined according to best intended meaning (for example, if one user created a category called “using VPN” and another “VPN connectivity,” the two categories were streamlined into one for purposes of comparative analysis.)

This card sort occurred prior to the performance of tasks (as described in Sapienza 2008) and hence, the results reflect user topical preferences without prior exposure to the actual site itself during the test session. This was done in order to best determine how users would organize the information without benefit of seeing how the site actually organized the items. However, a small portion of the users indicated exposure to the website prior to the study (see Sapienza 2008) so it is not entirely the case that all users were operating from a naive perspective about the site. Card sort results were compiled with the USort and EZCalc freeware applications by using the “average linkage” method.

Dendrogram charts show much similarity between the two groups of participants. The international participants generally divided topics into three categories, and the Americans divided topics into four categories. Within-category content of topics is relatively similar between-groups. This similarity is visually discernable through dendrogram summary graphs below:

American Participants
International Participants

Comparing Card Sorts Between US And International Users

The edit distance metric developed by Deibel, Anderson and Anderson (2005) was used to provide a more precise numerical measure of similarity between the two populations. This metric is similar to the Levenshtein method of string comparison in computer science algorithms, but the method is applied to two-dimensional matrices. The method measures the minimum number of moves needed to make two different matrices identical using insertions, deletions, and replacements of items. The edit distance can range from 0 (identical card sorts) to n (the total number of cards sorted, reflecting a completely different set of card sorts).

The average edit distance between international and American participants was 3, indicating that only 3 moves were needed to make the card sorts between groups identical. The edit distance between the American participants and the actual topical organization on the website was 4, whereas for international participants, the distance was 3, suggesting that the university web designers provided a site that may have come closer to approximating the international user student population than for that of American students. Rather than being marked by sharp divergent curves, the close proximity of the dendrogram summary graph further reinforces the similarity with which both groups organized the information for the site.

Dendogram Summary Graph


This brief part of the published study focused on the card sort results of international and American participants and found the difference between the two groups to be relatively small (3 moves). Further research would need to examine why these results occurred, and the degree of differences by culture of origin. Possibly, the university IT staff responsible for web development and maintenance is comprised of individuals originally from outside the United States. The institution comprises 3 separate campuses located in a major urban area with a high immigrant and transnational population.

The close results might also have to do with the nature of the user participants, which compared 24 people from one culture to 24 from several other cultures bunched together. Hence, edit distances between card sorts might normally be expected to be high between iAmerican users and users of a single particular culture. When bunched with other cultures, the results may be flattened out somewhat. Insofar as the purpose of a web development project is to create a web portal for all nationalities bunched together under one roof, these research limitations may be irrelevant. Without the benefit of a high budget, there is simply no way that any organization can create a portal for every single culture. Future research is needed to compare edit distances between two (or more) distinct groups.

Finally, the edit distance method does not take into account the multi-tiered architecture of a sort (that is, how users subcategorize topics) but rather, treats all sorts as first-level groupings as it is only concerned with clusters of similarly grouped topics.


Deibel, K., Anderson R., and Anderson, R. (2005). Using edit distance to analyze card sorts. Expert Systems 22 (3), 129-138.

Sapienza, F. A Shared Meanings Approach to Intercultural Usability. IEEE Transactions on Professional Communication 51 (2), June 2008, 215-227.