Social Software Evaluation Study

A study of tagging using Del.icio.us

  • Exploring sites and tagging them
  • Exploring others' tags
  • Reflecting on the process
This study ran between March and May 2007 and is now completed. We wish to thank the participants for the amount of time and effort they expended to help us with this.

This study explored the utility of tagging and folksonomy for transdsciplinary research. There is a focus on narrative in this study as a means to provide a common framework for all participants but in theory many other frameworks could have been chosen.

This was a speculative exercise. We were not working to a hypothesis, rather we were setting up a series of tagging and reflection tasks with a cadre of volunteers from a diverse background with a diverse range of expertise in tagging.

The experiment was divided into three stages. During the first stage of the experiment, each participant tagged the urls by themselves. In the second stage of the project, the participants collaborated to revisit some of their tagging choices. In the final stage of the project, participants were asked follow up questions.

When setting up the experiment we had two main approaches to choose between. In one of them, the participants would be asked to trial several social bookmarking services by tagging the same sites in each one and deciding which they preferred. The other option was to stick with just one service and tagging a greater number of sites. We chose the latter because user interfaces change from month to month and we were not so much interested in determining which was the best interface as we were about seeing what happened when diverse group of participants used the same software to tag the same sites. We chose del.icio.us simply because it was, at the time, the market leading social bookmarking service and it had good systems for exporting tagger data.

Non-technical Findings

There are three main conclusions arising from the tagging study.

Firstly, the taggers used significantly more terms as tags than they strictly needed to if they were using a taxonomic thesaurus. In total, the 30 participants used 1396 unique tags to describe the 40 websites. When this dataset was cleaned for obvious misspellings, variations in capitalisation and punctuation and so on, we were left with 1152 unique tags. We then compared this folksonomy with a standard thesaurus (Word Net) and were able to identify 721 unique WordNet words that mapped onto the folksonomy. Thus we can say that the folksonomy used approximately 60% more terms than an official taxonomy. This is roughly in line with other research thus our study helps verify pre-existing research into folksonomy.

Secondly, through analysis of the results from a survey of tagging exercise participant, we were able to ascertain two basic motivations behind tagging. We were able to infer this from a mixture of analysing their tagging patterns and their responses to our survey. The minority of taggers can be described as “indexers.” These taggers were more likely to clain that they “tag for others” and when tagging they were more likely to use a small number of tags repeatedly. Conversely, other taggers can be identified as “describers.” These taggers primarily claimed to “tag for themselves,” used a much larger number of tags than indexers and tended to reuse their tags less often. Their aim appears to be to use tags to describe the resources without worrying about consistent categorisation. Clearly any implementation of a folksonomic system will need to be able to cater to both types of user.

Finally, we discovered a strong resistance to “best practice” or any form of “standardisation” among the taggers. The vast majority of respondents who expressed a preference (19 out of 24) wanted no standardisation. However those who did express a preference for standardisation were those who tended to use tagging to “index” rather than describe. Much discussion about implementing folksonomy tends to focus on trying to make it work more like a taxonomy (e.g. by trying to develop pre-set categories) yet it seems from our study that participants strongly favour being able to tag without constraints. This appears to provide empirical support for the hypothesis that folksonomy will be most successful when users have suggestions rather than constraints.

Useful Resources

To get a flavour of how the study unfolded, check the TNN blog in general or the tagging experiment category on the blog. The sites used can be found on this website and also on del.icio.us at http://del.icio.us/tnn/ait. Other resources available on this website include:

Contact: Prof. Sue Thomas, School of Media and Cultural Production, Clephan Building, De Montfort University, Leicester LE1 9BH, UK. Tel. +44 (0)116 207 8266, fax +44 (0) 116 257 7199, email Sue.Thomas at dmu dot ac dot uk. http://www.ioct.dmu.ac.uk/tnn/