Thursday, October 4, 2012

Cheap & Cheerful Approach to Analysing Tracking Data

At the Museums Australia conference last week, I presented a paper with Jenny Parsons from the South Australian Museum about how the museum used volunteers to collect tracking data and the method I used to analyse the data and get a highly visual 'hot and cold spot' map of the gallery. The slides have been posted to slideshare:

The abstract of the paper is as follows:

In 2010 the South Australian Museum Foundation undertook to update the Australian Aboriginal Cultures Gallery (AACG) of the Museum. Originally opened in March 2000, the gallery was in need of updating in terms of technology, signage and interpretive material.

Without the benefit of a multi-million dollar budget, the project team had to be quite strategic on the edits and adaptations to be made. The team spent a year examining the strengths and weaknesses of the gallery and conducting strategic visitor evaluation.

As part of the evaluation, volunteers from the finance firm JBWere tracked over 150 visitors as they moved through the Australian Aboriginal Cultures Gallery (AACG), using a methodology that had been adapted from one used at the Detroit Institute of Arts. The employees undertook this as part of their corporate social responsibility efforts and so were unpaid. The museum felt that these analysts were ideal candidates for this evaluation as they were astute at monitoring and understanding data.

The purpose of the tracking was to find out how visitors moved through the gallery - which way they turned upon entry, which displays were the most visited, which sections were being missed, and how long visitors spent in the spaces overall.

Adapting methods from the visitor research literature, recorded visitor movements were coded and quantified using Microsoft Excel. This data was re-presented in a colour-coded format on a plan of the AACG. This relatively simple approach was able to create a highly visual and intuitive interpretation of the data, showing visitor movement patterns at a glance.

While the visitor tracking confirmed some of the exhibition team’s suspicions about how the space was being used by visitors, some of the findings also challenged assumptions and led to a revisiting of the way important orientating material was displayed in the redeveloped AACG.

Any questions about the method?  Feel free to drop me a line at enquiries [at] 

KISS program evaluation

Short and sweet program evaluation has five measures.
  • First time vs repeat use
  • Rating - 5-point scale and/or Net Promoter Score
  • Rating on KPI outcomes
  • Open comments
  • Demographics - age, sex, location (local or out of town)
I presented a snapshot at the Museums Australia 2012 conference about short and simple program evaluation. Here is the presentation.

To see the notes with each slide, view it on Slideshare by clicking the Slideshare button to the left of the arrow buttons. Then you'll see two tabs beneath the slideshow - one for comments and one for notes for each slide.  Click the 'notes' tab to see them.

Once you have collected the data, you'll need to tally it up and write a short report.

You can keep track of visitor experiences by implementing this really simple evaluation tool for all your public programs.

Author:  Gillian Savage.

Friday, June 8, 2012

Evaluation: it's a culture, not a report

The UK Museums Journal website has recently published the opinion piece Why evaluation doesn't measure up by Christian Heath and Maurice Davies. Heath and Davies are currently conducting a meta analysis of evaluation in the UK.

Is this the fate of many carefully prepared evaluation reports?

The piece posits that: "[n]o one seems to have done the sums, but UK museums probably spend millions on evaluation each year. Given that, it’s disappointing how little impact evaluation appears to have, even within the institution that commissioned it."

If this is the case, I'd argue it's because evaluation is being done as part of reporting requirements and is being ringfenced as such. Essentially, the evaluation report has been prepared to tick somebody else's boxes - a funder usually - and the opportunity to use it to reflect upon and learn from experience is lost. Instead, it gets quietly filed with all the other reports, never to be seen again.

So even when evaluation is being conducted (something that cannot be taken as a given in the first place), there are structural barriers that prevent evaluation findings filtering through the institution's operations. One of these is that exhibition and program teams are brought together with the opening date in mind, and often disperse once the ribbon is cut (as a former exhibition design consultant, their point about external consultants rarely seeing summative reports resonated with my experience). Also, if the evaluation report is produced for the funder and not the institution, there is a strong tendency to promote 'success' and gloss over anything that didn't quite go to plan. After all, we've got the next grant round to think of and we want to present ourselves in the best possible light, right?

In short, Heath and Davies describe a situation where evaluation has become all about producing the report so we can call the job done and finish off our grant acquittal forms. And the report is all about marching to someone else's tune. We may be doing evaluation, but is it part of our culture as an organisation?

It might even be the case that funder-instigated evaluation is having a perverse effect on promoting an evaluation culture. After all, it is set up to answer someone else's questions, not our own. As a result findings might not be as useful in improving future practice as they might be. So evaluation after evaluation goes nowhere, making people wonder why we're bothering at all. Evaluation becomes a chore, not a key aspect of what we do.

Friday, April 27, 2012

DIY Curator

Most social media platforms make it easy for users to select images and present them to the world. Users can collect their images and annotate them, then make selections, curate an album and give it an overarching story.

Flickr, Facebook, Tumblr, blogs, and any number of variants allow everyone to be a curator.

The newest kid on the block is Pinterest, a content sharing service that allows members to "pin" images, videos and other objects to their pinboard. With 2.2 million active daily users and 12 million active monthly users, Pinterest is now the third most-used social media platform in the United States.

Most major museums have adopted social media by using blogs, Facebook, Flickr, etc. How soon before they start colonising Pinterest?

The basic contract between these social media sites and their users is that users need not be passive consumers, they can be the active party, going beyond just looking or reading to 'like', share, recommend, comment, re-post, and add their own content.

Pinterest is popular because it feeds our appetite to share the things we’re passionate about. It has all the buzz of the latest thing. Forbes journalist Scott Goodson wrote, somewhat breathlessly,
Pinterest drives more traffic than three of the biggest social networks combined. And you tap into a digital world of curation, sharing and visual inspiration. It’s the future and it’s happening right now.

Museums that allow visitors to be active and respond socially and creatively in a variety of ways will be loved as much as these gorgeous, functional and popular social media platforms.

Wednesday, February 29, 2012

Committee is at work

Your EVRNN Committee met by teleconference yesterday. We polished off a few admin tasks in quick fashion. You'll see that our first action was to fall into line and accept a name change in line with MA's networks. So goodbye EVRSIG, and hello EVRNN.

Then we got down to discussing some of the things we could offer at the MA Conference in Adelaide on 24-28 September 2012.

We are keen to encourage you to offer papers about any aspect of visitors or audiences at your place. The call for abstracts end on 27 april 2012. If you'd like to discuss an idea for a paper, or seek a co-author, please contact a committee member and we'll see how we can help.

In addition, EVRNN would like to offer something along the lines of hands-on workshops for beginners and/or advanced practitioners.

And we'd like to shape up a fun interactive session that would use active learning principles rather than the 'sit and listen' approach that is the usual fare at conferences. What form could it take?
  • Quiz, trivia or puzzles to solve?
  • Debate or competition of some kind?
  • Mythbusters?
What do you think?

And, now to end with some eye candy... here's a picture from the Canning Stock Route at the Australian Museum. I blogged earlier about this fabulous exhibition when I saw it at the National Museum. It looks just as good at the Australian Museum. 

Canning Stock Route - sinuous bench with built-in interp

Posted by Gillian Savage.

Tuesday, February 21, 2012

2012! What already?

Since I first took up the EVR position at Museum Victoria in January 1994 much has changed in regards to EVR within my organisation and in the museum sector.  EVR practitioners have become more skilled and professional in conducting studies and we continue to experiment with new methodologies.  Between us all we have probably evaluated every aspect of the work of museums and their engagement with the public. And now we have large storehouses of studies, and insights from them, in our organisations – I have just completed the 833th study conducted here at Museum Victoria.  EVR no longer needs to justify its existence and people within our museums naturally turn to it to inform their work.

However, in recent times I have begun to wonder if that is about to change.  Has the visitor’s voice become so central to our thinking that we run the risk of believing we know it all and so do not require EVR as we have in the past?  In fact, fewer full-time EVR positions exist and there is a greater reliance on part-time positions and student projects.  Indeed, it is more likely that EVR is represented within an organisation by someone who is keenly interested but has another role to fulfil as well.  I have also noticed that the practitioner is being replaced at conferences and the like by an increasing number of academics.  So are thing changing and what might be the future for EVR in the museum sector?

I asked Dr Patrick Greene, CEO Museum Victoria and the new Chairperson of the CAMD, his thoughts on the past and future contributions of EVR to the sector.

On arrival from Manchester 10 years ago, Dr Greene was impressed by how museums in Australia had embraced EVR to inform their work.  His experience in Manchester confirmed for him that evaluation most definitively leads to better exhibitions and even now it is important to keep asking public what it is they want from museums.  Dr Greene stated that EVR continues to be a strong factor in shifting the long standing belief that museums know best to one that recognises that people in the community have all sorts of valuable knowledge that museums must embrace.

 As for the future, the new frontier is digital and with the NBN roll out it is particularly pertinent.  Museums are faced with a bewildering range of opportunities for using digital mediums, but we do not yet know which ones will work best for us.  Unless we become deliberate in examining the effectiveness of each form we won’t know where to put our effort. The rate of change is so speedy, we find ourselves in a scramble to catch up.  We must get the right tools in place to do work within our organisations and we should look to collaborating across museums.  

 EVR also has a strong role to play in providing information that can be used to advocate the work of museums to stakeholders and partners.  The assertions of an organisation are all very well but they are made stronger when backed up with facts which add authority and verification.  If museums are not already involved in advocacy they should be, and it will only become a more important thing to do to keep our museums sustainable.  This work, and the role of EVR in it, is ongoing.

 So if there is, as Dr Greene says, a strong need for EVR in the sector, how can the EVR SIG support this?  There has been a slow demise in the vigour of this, and other, SIGs in recent times.  Member numbers have declined and contact between them has become tenuous.  At the next committee meeting, we will be discussing the role of the SIG.  Should we reinvigorate it or has it no role to play? Use the comments section below to tell us what you need from your EVR SIG.  The committee will get back to you with some initiatives either way. 

A rose by any other name . . . 
Our name, Evaluation and Visitor Research Special Interest Group (EVR SIG) needs to change in line with Museums Australia renaming of these dedicated interests groups.  So the new name is Evaluation and Visitor Research National Network (EVRNN). How do people feel about that?

Caroyn Meehan
EVRSIG President