Measuring Impact: open space at Agile Iceland

I recently spoke at and attended Agile Iceland in Reykjavik, a fun, friendly and inspiring conference, hosted by the capable crew at Kolibri. It’s a one day event, but they devote about a third of the schedule in the afternoon to an open space. If you’ve been to a BarCamp then the format is very similar. The open space sessions are entirely participant driven and they provided the opportunity to discuss ideas and topics that came up earlier in the speakers’ talks and workshops.

A grid is put up on the wall and participants come forward with session ideas written on post-its, introduce their topic and post them on the grid. Here’s Dadi from Kolibri explaining introducing the open space and explaining a couple of rules:

Kicking off the open space

The board soon filled up and once we’d figured out what interested us most, we all trotted off to different sessions.

Open space session grid

The second session I went to was titled ‘Measuring Impact’. I found the open space session really helpful so thought I’d write up a quick summary.

Some inspirations for this session from talks earlier in the day were:

  • Mary Poppendieck’s keynote about the Lean Mindset and impact-driven development
  • Rich Smith’s talk about culture at Etsy and their principle “If it moves, graph it!”

Pétur, who proposed the session, wanted to find out more about measuring impact. He had worked on a project where they measured a lot of things, and collected a lot of data, but didn’t know which metrics were important. So consequently they stopped looking at the data. Sounds familiar? Well it did to me.

The group agreed that the most important question to start with is why? Too often people try to measure everything, without thinking about what the right metrics might be. To be sure you’re measuring the right things, you need to think about what impact you are trying to measure.  A good technique for understanding what you need to measure, and why, is impact mapping. I’ve not used this technique before, but there’s an example explained here.

In the case study which inspired the session, a Telco company wanted to allow customers to self-serve to reduce costs. They also wanted to improve customer experience. I talked a little about my experience of measuring the cost of a transaction, sometimes called the ‘cost to serve’ when I worked in local government. I explained how we used standard transaction costs for customer contact by phone, face-to-face and web to determine the reduction in overall transaction cost. I showed the GDS service performance dashboards, which I think are a great example of monitoring the impact of service design changes.

We talked a little about the different contexts in which a customer might be driven to use self-service over mediated service channels. Someone gave an example of a petrol station in Iceland that is entirely self service but is rated as one of the best petrol stations. We also discussed the fact that your customer might be forced to use the product. So it’s important to measure customer satisfaction. A measure that can be used to track satisfaction rates is the Net Promoter Score ®, where customers are asked how likely they would be to recommend the product or service to someone else.

Net promoter score

 

Although product teams can stress over getting the right metrics, we discussed the fact that you don’t need a perfect measure to start with, but it’s important to iterate and learn as you go.

Mary Poppendieck advised that it’s helpful to show progress by making metrics visible. It’s a great motivator for teams working on a product.

A couple of people gave examples of when measuring impact had worked for them.

One person described how, when launching a new system, he analysed the number of transactions and monitored errors via the logs. He called customers up if he saw errors during their sessions to find out more about what had happened. Although this seemed a little creepy at first, the customers were more than happy to talk about their experiences using the website.

Sometimes you can measure what Eric Ries, author of the Lean Startup, calls ‘vanity metrics‘ rather than metrics that show something meaningful about how customers use your product. Someone described how their company measured usage going up but then also noticed their sales were going down. They carried out some usability testing and found out that registration process was far too complex and hard to use, so they simplified it and sales started to increase again. In this example focusing on usage alone was unhelpful.

Although we only scratched the surface, this was a useful discussion and I learned something new. In summary the key takeaways from this discussion were:

  • Start with asking why, to find right metrics
  • Create an impact map, to understand what to measure and why
  • Iterate the measurements, and improve as you learn
  • Make measurements visible, to motivate the team
Share

Why I like sketchnoting

If you’ve attended a UX talk or conference recently, the chances are you will be very familiar with sketchnoting, the art of making visual notes whilst listening to a speaker. Some of the better known sketchnoters are Eva-Lotta Lam and Mike Rhode.

I was an avid sketcher as a child and used to spend at least a couple of hours a day sketching, using my family and our pets as subject matter. I studied art as part of my degree at University, but sadly stopped sketching when I started working in the administrative side of the creative industry in the mid-1990’s. I’m not sure why this was, but I guess sketching wasn’t something that was encouraged in the jobs I was doing at that time. And when I started using the internet my new found love of all things digital soon overshadowed pencil and paper. But over the years I have continued to doodle in my notebooks, particularly in boring meetings.

Last year I changed my career path and started working in a dedicated user experience role. Because sketching is part of the UX design process, I was keen to improve my sketching confidence and ability. I also organise talks for a local UX group and used to find myself making lots of written notes, which I never referred back to.

I attended my first UX conference in November 2011 and armed with a sketchbook, decided to give sketchnoting a go. These are a couple of examples of my first attempts at sketchnoting:

 

I was fairly happy with the results but wanted to improve, having seen so many great sketchnotes online. Since the end of last year I’ve learnt a few practical things about how to improve my sketchnotes.

  • I now use a bigger sketchpad (A4) as  it makes it easier to fit one talk per page.
  • At first I felt under pressure to limit the text on the page, but now I find that adding notes with images helps me remember the talk better, and also reduces the pressure of producing a page of images.
  • Using a shading pen (e.g. a Promarker warm grey 3 or 4) helps to give people and objects form 3-dimensions.
  • You need to think about how to represent the flow of a talk – this is quite hard to achieve, but over time you get a better feel for it. Using arrows and dividers can help.
  • Coloured Promarkers provide a vibrancy and balance to the sketchnotes but they do tend to soak through paper, so it’s best to put a loose sheet of paper in between the pages to avoid marking the next page.
  • Practise helps. A lot. I attended a very useful session led by Eva-Lotta Lam at UX Camp London on how to draw people. She gave us some useful tips on drawing hands and bodies. But I know I just need to practise more!
An important lesson I’ve learned is that if you make mistakes, don’t sweat it! It happens. Just improvise and you’ll find ways to turn them into something that isn’t that discernible to others. For example adding some shadow to cover the mistake. Many of the people in my sketches have ended up with two elbows, or mis-shapen heads, which reminds me of the character Ushikawa in Haruki Murakami’s wonderful trilogy 1Q84.

I find that sketching helps me concentrate and process more information. An academic study in 2009 actually provided some evidence for this hypothesis. This hypothesis is further supported by Paivio’s Dual-coding theory, which proposes that it is easier to recall a piece of new information when we process images and words that represent it.

In general my ability to recall factual information from memory is not great. I have what people refer to as a visual memory. At school I could recall information I learnt in lessons if I could visualise a doodle I’d made in the margin of my notebook. Unfortunately I got into trouble at school for sketching in the margins of my notebooks. I just wish my teachers had realised the benefits! The actual process of sketchnoting helps me think spontaneously of visual representations for information, which in turn helps cement information in my memory in a way that is easier to recall.

Sketchnotes help me concentrate, absorb and process information and recall it afterwards. If I wasn’t sketchnoting I’d probably drift off, especially when I’m sitting in a full day of talks at a conference. But I also enjoy sharing them and it’s fun to compare sketchnotes with someone else who was at the same talk.

Here are a few personal favourites from my Flickr stream:

 

 

You can see loads of inspirational sketchnotes from around the Web on Sketchnote Army, a website set up by Mike Rhode.

 

 

 

Share

Doing rapid product design in the wild

Last month I was lucky enough to try out a completely new approach to running a stand at a software tradeshow, in Texas. Over 3 days we designed and developed a prototype for a new software tool. Using rapid prototyping techniques (paper and HTML/CSS prototyping) and Agile development methods, we were able to get feedback from prospective customers and iterate our designs during the tradeshow.

Our stand was specifically designed to provide a UX area for working on designs and gathering customer feedback, and a development area with a Kanban board to track our progress. We made the process completely transparent by putting up all the feedback we received on the walls of the stand.

I’ve written a case study of the ‘Live Lab’ concept, describing the process we used, on the Red Gate UX Blog. I’ll also be writing up the lessons learned in a separate post very soon.

In September I’ll be doing a talk on this case study at Agile Cambridge 2012. I did a warm up talk at UX Camp London last Saturday:


 

 

Share

Don Norman advocates systems thinking in education

In a recent 3 minute video interview with the Illinois Foundry, Don Norman advocated a systems thinking approach to design engineering education. Rather than educating students in narrow, deep specialisms, Don suggests that Universities also need to educate students with a broad knowledge of social sciences. Educating systems thinkers will, he feels, ensure user needs are considered and help solve social problems.

We need people whose understanding goes beyond science and engineering to encompass history, literature, and the arts. We need people who understand both technology and people.  The emphasis on specialized knowledge is bad for society.

Share

Design principles

I’m doing some research into design principles for a project at work, with some colleagues. Design principles help focus and guide the design process to align product development with organisational objectives, brand values and user needs. Design principles provide strategic direction for design work, particularly for larger organisations with a number of products or services. But they can also work for start-ups and smaller organisations, who want to ensure they don’t stray from a clear vision that makes their product or service unique.

Design principles can also be thought of as best practise heuristics. I’m not going to write at length about how design principles can be used, as I’ve found some really useful resources elsewhere. For example Whitney Hess’s presentation, and posts by Jax WechslerLuke Wroblewski and Sarah Nelson.

There have been a couple of other notable posts recently which seem very timely given our design principles project. The first was by the UK Government Digital Service who have published a set of 10 design principles for developing Government digital services. It’s inspiring to see that GDS are using principles to focus their design projects. I particularly like the way their principles are written: clear, concise and no-nonsense. Juksie’s post about the GDS design principles is well worth a read.

The second post which came to my attention was from Abbey Covert, who has developed some IA heuristics. Abbey’s approach to developing the principles was led by reviewing the existing literature on IA best practise principles and heuristics, and synthesising her findings into a new set of heuristics.

One of our starting points has been to review existing design and usability principles, developed by UX researchers and other well known organisations. To help understand which principles are referenced most frequently, I collected principles from various sources (listed below*), added them to a spreadsheet, categorised them and visualised them as a word cloud:

A word cloud of frequently used design principles

I had to make some subjective decisions about how to categorise the principles, but it was fairly clear which are the most popular ones e.g. ‘consistent’, ‘simple’, ‘user control’, ‘inclusive’, ‘feedback’, ‘aesthetic design’. All these principles were included in Norman’s design principles and Neilsen’s heuristics which indicates that many organisations may have referenced those sources. It isn’t surprising that these existing principles have propagated, because they are are borne out by best practise and research.

‘User centred’ was mentioned less frequently in the design principles I analysed, perhaps because it seems too obvious. Newman and Lamming (1995) highlighted the importance of including even the most obvious principles:

The need to design with a view to supporting human activity is so basic it often gets left out of people’s general principles or ‘golden rules’. So it needs to be stated here at the outset:

  • Design with a view to supporting the user’s task or process.

A second principle relates to the need to address the concerns of the user. If we know who this user is and have some familiarity with his or her special needs, we can orient our design strategy accordingly. But we often lack this understanding. A fundamental guideline to follow is Hansen’s ‘user engineering principle’ (1971):

  • Know the user.

By a combination of studying the user’s activities and learning about their skills, knowledge, roles and responsibilities, we can design according to these two principles.

I was also surprised that references to emotional responses such as the principle ‘delightful’ were relatively under-used. Especially in a time when there seems to be a proliferation of books, talks and conference workshops covering topics such as designing products with personality, or designing for emotional engagement and delight.

Design principles can be developed at different levels. They can be universal, covering the entire design output of an organisation. Or they can be tailored for specific products or projects. We are aiming to develop some universal design principles in the first instance.

As we move to the next stage of developing our design principles I’m interested to know how well design principles work elsewhere. Does your organisation use design principles? Are you willing to share them? How have they worked for you? Do your products adhere to the principles? What happens if a design principle is ignored? Is there any governance to ensure design principles are followed? Please leave a comment if you have any insights to share!

* The principles we have referenced so far include:

Apple iOS
Bruce Tognazzini
Facebook
Government Digital Service
Google
IBM
Morville, Peter
Nielsen, Jakob
Norman, Don
Schneiderman, Ben
The Understanding Group’s Heuristic IA Principles
Windows

 

 

 

Share

Designing a (better) civic future

I was really excited to come across a recent post from Analisa Lono at Adaptive Path about a new course they are putting together with the University of San Francisco:

Our goal with this course is to provide the opportunity for students to gain exposure to concepts, methods, and professionals outside of their academic disciplines while getting first-hand experience creating a digital product for the purpose of serving their community.

Young Rewired State and Apps for Good are two inspiring UK initiatives that encourage young people to use digital technologies, to provide innovative and useful solutions to social problems. Young Rewired State, a spin off from Rewired State, follows a hack day format and is focused on developing coding skills. The founder, Emma Mulqueeny, also co-founded the Coding for Kids movement, to campaign for a more relevant ICT education that gives young people the ability to shape their digital future. Apps For Good is an initiative for young, unemployed people which focuses on developing entrepreneurial and creative skills. The participants also learn practical, iterative design techniques such as paper prototyping.

 

Participants in the London Design Jam

Photo from London Design Jam 4, by Joe Lanman

Design Jams are similar to hack days and usually take place over one or two days. They provide an opportunity for less experienced UX practitioners and students to work with mentors on solving actual design problems. Some of the design problems tackled have been civic issues, for example Design Jam Oxford tackled the perennial problem of how to improve the experience of commuting with digital technologies.

Similar to Design Jams and Charity Hack days, again with a practical design focus, Cooper Design recently ran a UX Bootcamp which enabled students to try their hand at designing mobile apps for the Red Cross.

There are two things that interest me about the new course in San Francisco:

  • Firstly that UX practitioners will mentor the students, similar to the Design Jam and UX Bootcamp events mentioned above. Adaptive Path are teaming up with Seabright Studios and Grey Area Foundation for the Arts to develop and run the course. And the course will be rooted in design practise. Having studied a Master’s degree in HCI, I feel strongly that HCI and Interaction Design courses should be equally taught by design practitioners and academics.
  • Secondly, the course is dedicated to exploring how great design can solve civic problems. Hopefully this means that there will be a new generation of designers who are inspired to use their skills for social good.
Share

Rahel Bailie – The Content Strategy Paradox

On Monday the Cambridge Usability Group welcomed Rahel Bailie, all the way from Vancouver, to give a talk titled ‘The Content Strategy Paradox’. There’s been a lot written about content strategy recently. I’ve read articles and books that promote content strategy (too many to list here) and some articles that question the existence of content strategy as a standalone discipline, like this recent one. So I was keen to hear what Rahel had to say.

The first part of Rahel’s talk addressed confusion over the definition of the term ‘content strategy’. She highlighted that recent workshops and events have promoted a number of seemingly paradoxical interpretations of the term ‘content strategy’. But in reality, whilst there are a wide range of specialisms that involve content (including web, social, mobile, editorial, brand positioning, marketing, support and technical), there is only one discipline. In other words, it’s all content strategy!

Rahel then covered some of the processes and approaches to researching, planning, developing, publishing and managing content.

One of the main takeaways for me was that “Consumers use all types of content to make buying decisions”. Customers don’t differentiate between marketing, support and social content, so neither should you.

Rahel provided some excellent ‘do’ and ‘don’t’ examples of how content affects our purchasing decisions. I think everyone who uses the internet has experience of struggling to gather and compare product information before making a purchase. Companies that fail to focus their content on user needs and tasks or provide sufficient, detailed support information and leverage user generated content are missing a trick!

However, having great quality content is not enough. Rahel outlined that you also need to know how and when to deliver your content, and to which audience. Think about whether all your content is relevant to all your customers. Chances are it isn’t. This tweet from one of the audience summed up the amusing example Rahel gave us!

Overall it was a really entertaining and useful talk. I was busy sketching away throughout (something I’m trying out to improve my skills in sketching and summarising information), so here are my sketchnotes:

Sketch notes from Rahel Bailie's talk on Content Strategy

And here are Rahel’s slides from a previous presentation of the same talk:


Share

Video of my #uxcam talk on embedding user experience

The talk I presented at UX Cambridge 2011 was filmed by InfoQ and is now live on their site, together with my slides:

Winning hearts and minds: how to embed user experience from scratch in a large organisation

The talk is about 45 minutes long and includes several lessons learnt and the findings from a survey I conducted on UX maturity in other organisations. I also wrote a blog post summarising the talk last year.

Screenshot of my UX Cambridge talk published on InfoQ.com

 

 

 

 

 

 

 

 

 

 

 

 

Share

Adjusting to a multitude of changes

I’ve been in my new job for almost three months now and I’m really enjoying the new challenges. Anyone who reads my blog will know I used to work as a Web Strategy Manager within a local authority.

Towards the end of last year I switched to a User Experience Specialist role in a software company. I was keen to develop my UX skills and an opportunity came up at Red Gate, a local software company who are really committed to making sure their products have good UX. So much so, the company strapline is ‘ingeniously simple tools’!

It’s been a big change for me in many ways:

  • I moved from a management / strategic role to a more hands on role.
  • I moved from a generalist role to a more specialist role.
  • I moved from the public sector to the private sector.
  • I moved from a role focused entirely on Web, to one that’s mostly focused on desktop (Windows) software.
The Web sites I worked on previously were informational and transactional. The products I work on now are essentially productivity tools. Windows applications that help people work smarter, faster and reduce errors when carrying out complex tasks. The tools are aimed at database developers, so it’s also a fairly technical domain.
There have been quite a few changes in how I work. Here are a few of my observations.
  • In my last job we had a very wide range of users. Now I deal with a niche group of users. This is actually easier in a lot of respects when you are recruiting customers for interviews or usability tests.
  • For Windows software I’ve found that pattern libraries and UI examples seem few and far between compared to Web exampes. Maybe I just haven’t found the right resources yet. Our UX team still tends to take a lot of inspiration from Web and mobile apps and I’ve discovered sites like Quince which are helpful additions to the Window guidelines.
  • It can be harder to release frequently and gather usage statistics for A/B tests. I realise now how much I took Google Analytics for granted! But our team have been working hard to address this issue.
  • I’m designing for rich interactions much more than I ever used to. For example the software I’m currently working on supports people to write code in a development environment. I think there’s a kind of ‘UX zen’ you need to achieve when designing this kind of software. It requires close observation of user behaviour and continual contextual enquiry to understand user goals, context of use and empathise with their pain points.
  • In my previous role our customers didn’t have much choice about using our Web site. Now I work on products in a competitive, commercial environment. Our products have to be useful and easy to use and have a ‘wow’ factor, which creates a loyal and happy customer base. I don’t believe this means I think about the User Experience in a different way, it just means there’s a different set of business objectives and constraints to consider.
  • Although we used the Scrum development method in my last organisation our sprint progress was tracked with a burndown spreadsheet, due to lack of wall space. This used to frustrate me, so I’m now really appreciating the extensive use of whiteboards and stickies! Pretty much all the work the team is doing and our progress is very visible on the walls around us.
  • Although I don’t manage people directly, I still take an active interest in our team’s development and recruiting new staff. So hopefully I can make sure my management skills are still being put to good use.
  • Last but not least, I am rubbish at writing SQL queries! But trying to improve.
I’m learning a huge amount each week. I’ve taken a bit of a break from this blog to adjust to the changes. But I’m hoping to get back to blogging now that I’ve settled in and everything is becoming more familiar.
Share