- Recommended book: Letting Go of the Words by Ginny Redish
- “Can a usability test just use a questionnaire, no observation?” (Audience: you need to see what they do, not just what they say; observation gives us context, avoids bias)
- Surveys have inherent bias – people will say what they think you want to hear
- It’s not a usability test unless it’s got some observation in it
- The more interesting question is: Can we use surveys to assess usability of our websites?
- Survey data can aid by triangulating between survey data and data from elsewhere (e.g. user testing)
- Surveys can assess: effectiveness at task completion; satisfaction; demographics (stuff that can’t be tracked with analytics)
- What is “the product” that we are assessing? Is it the website, a section, a page?
- Net Promoter Score (NPS) — “Would you recommend us to a friend” — expands ‘the product’ up a level to be ‘the brand’, but can be horribly muddied if you try to fudge the question
- Survey Tip: Ask one question at a time
- Best thing you can do with surveys is looking at goals: “what did you come to this website to do today?”
- Survey Tip: Find out about users’ goals
- There are other goals to consider: What the organisation wants to achieve, What are our aims in doing a survey
- Establish your goals for the survey (what are the questions you need answers to)
- Types of survey goals: Exploratory (don’t know what is out there, so we send and hope or use a website popup. If you don’t know anything, any answers are better than no answers, as long as you don’t try using them as descriptive statistics with them); Comparative (explore trends, compare before and after – great for looking at trends, but you have to use the same questions every time, therefore you have to get the questions right first time); Descriptive (you know what is out there, go and count them — e.g. national census — got to do this properly with balanced sampling to avoid bias and measuring error); Modelling (find factors that show cause and effect – you’re seeing behaviours, trying to find out why)
- You can combine types of survey – e.g. cohort studies use comparative, descriptive and modelling together
- Examples of poor survey questions: poor defaults, double questions (two questions stuffed into one), options that we should already know the answer to via tracking, use of jargon, grid of radio buttons (contributes to user dropout)
- Bad surveys can have a negative effect on brand
- Survey Tip: Interview first (talk to your users – good surveys always start with interviews, so you can understand how people want to respond to your questions)
- Survey Tip: A successful survey is a process that involves loads of testing
- Don’t focus on the idea, but the process of imagination/problem solving
- Imagination can be thought of as a Venn diagram: What has been, What is, What could be — in UX, this maps to Insight, Empathy, Creativity
- Creativity. Compare Turner’s paintings from the start and end of his life; what happened in between to change him? He was apprenticed (learned from experts), thought about colour theory, using new technology (new colours became available, tubes of paint were invented), travelled (UK, Italy, France, Switzerland) to learn about differences in light/location, produced over 19,000 works.
- The baker Christina Tosi: started with lots of experimentation and lack of constraints.
- Showed a picture by Felix Baumgartner aged 5 of himself falling from the sky; he had a focus, having a vision is powerful.
- Becoming imaginative isn’t passive, it’s something we can consciously do. We can practice what we do to build up our skills. Teaching helps us learn how to communicate our ideas to other people. Observation shows us what is going on in the world/people around me. The ability to Experiment/prototype; asking questions/debating and reflection – what can i learn from what i’ve done so far. Vision – know the direction in which you are heading.
- Insight + Empathy = Deep understanding, which comes from better research. It’s very easy to get into the habit of doing a little bit of regular research, which then just sits there. You’re not getting any value or understanding out of research.
- mobiserve.eu is a project investigating how we can keep the elderly population more active/mobile in their own environment. Involves a robot that interacts with a person. Researchers used disposable cameras, 2 per person, with a sad and happy face drawn on them. Users were asked to use the cameras to take photos of things that made them happy or sad. These photos (and the explanations behind them) increased designers’ empathy towards their users. Interviews are important to reduce assumptions about users’ feelings.
- How about our own experiences? Richard talks about having testicular cancer. Moved from basic empathy to deep understanding of what it feels like to have that kind of illness. Sometimes we forget users in research have multi-faceted lives. Imagination draws on your own experiences to heighten feelings of empathy and decisions you make.
- As designers we create and think we can solve problems.
- Arthur Fry worked for 3M and sang in his church choir, but he often lost his place in his hymn book. He wanted to invent something to keep his place, and the Post-It note was born.
- As humans we have the incredible ability to feel for people we don’t know. Need our stakeholders to have empathy for their users. Case study: a local council site with problems. It would have been easy to just make the buttons and text clearer; instead used empathy mapping techniques to go through planning application process. Natural empathy makes people build more detailed picture of user than just listing facts.
- In the 90s, a virtual reality experience was used to give doctors an experience to encourage empathy for patients suffering from fatigue. 60% of doctors said they would change how they would treat their patients.
- What about our projects? Treat projects as several periods of open imagination closing in on solutions/answers. Consider the dynamics of projects, the context/level of detail, and activities to stimulate creativity.
- Someone to encourage, someone to challenge – we need people that fulfil these roles for us.
- Constraints vs freedom. As designers, constraints are more useful – they make us more innovative (e.g. car design has improved within the constraints of legislation around emissions).
- New and familiar – the temptation is to be really creative, but sometimes customers need familiarity. You need both.
- Play. The people we are designing for are also imaginative and playful people. They want freedom. If we try to constrain people when they’re in their decision-making process (when all the things affecting their decision are jumbled up as if in a washing machine) it feels unnatural. There is a massive experience gap between what people are trying to do and what technology is letting them do. Computation doesn’t fit with people’s imagination/individuality. We want people to think and imagine.
- An example of a dull product put across in imaginative way: an old Union Carbide ad.
- User research involving mobile devices gets much better/truer responses when users can use their own device instead of being given one.
- Book recommendation: Computers as Theatre by Brenda Laurel
- Story of the invention of the Brooks bike saddle. John Brooks was a leather manufacturer. His horse died, he couldn’t afford a new one, so he borrowed a friend’s bicycle, which back then didn’t have saddles (instead they had wooden planks). So Brooks patented the leather bike saddle. Bike saddles are interesting technology, because they get better with time (becoming more comfortable). How can we create stuff like this?
- “Be yourself, everyone else is already taken.” – Oscar Wilde
Sketchnotes of the session by Jenny Willatt:
The eighteenth Cambridge Geek Night took place yesterday in the august surroundings of the Student Union Society main hall.
After an introduction was recorded by the BBC World Service edition of Click, including a prompted round of applause that sent the youngest member of the audience crying from the room, the host for the evening introduced the three speakers.
Mark Henderson is Head of Communications at the Wellcome Trust, “a global charitable foundation dedicated to achieving extraordinary improvements in human and animal health by supporting the brightest minds in biomedical research and the medical humanities.” He is also the author of The Geek Manifesto: Why Science Matters, a book concerned with placing science more firmly at the front of political decision making.
Dr Julian Huppert MP was elected as the Liberal Democrat MP for Cambridge in 2010. He is the only MP to have previously worked as a research scientist, and is the author of a recent Lib Dem policy paper, Developing a Future: Policies for Science and Research.
Dr Emily Shuckburgh is a climate scientist. She is currently head of the Open Oceans research group at the British Antarctic Survey, a fellow of Darwin College Cambridge, and an advisor to the government Department of Energy and Climate Change.
(These notes are presented as summary. Please comment or email with corrections.)
- Talking about the cover of The Geek Manifesto, which is a pastiche of Communist-style art; the first design was rejected due to the illustration featuring a beaker of fluid not behaving according to the laws of physics (although many geeks supplied explanations as to why it might be correct).
- They also had to change the colour of the cover from blue to orange to avoid it being mistaken for The Greek Manifesto.
- Themes of the book: there is a disconnect between science (in the broadest sense, as advocated by scentists such as Carl Sagan) and how science is embedded into public life, which leads to policy failure again and again. Very few MPs have any track record in any science at all. This leads to a couple of problems – without experience with science, politicians and civil servants mishandle science. Without understanding of science, they fail to exploit science as intelligently or effectively as they could.
- We end up with things like well-meaning medical interventions (Human Tissue Act, Clinical Trials directive) with no understanding of the potential impact.
- This approach to scientific problem solving is also important where we don’t know the answer. We can use methods that sciences have developed to evaluate policy interventions a lot more thoroughly.
- For example, teaching kids to read. There is no evidence from randomized control trials whether phonics is the best approach to teach reading; in fact there is other evidence that points to different approaches being better. Phonics has been used now for over 15 years – if a proper trial had been set up then, we would know for sure by now.
- The other aspect of the book is the abuses of science by politics. The reason it gets mistreated is partly the fault of those of us who care about it greatly. Most politicians do not mishandle science because they mean to. The vast majority of MPs are not hostile – they have just not thought much about science. The idea of a randomized trial is anathema to them.
- There is a huge opportunity for the rise of geeks, and the way that popular culture now celebrates curiosity and science (cf. Ben Goldacre), to change attitudes to science.
- It’s a great privilege to follow Mark; he’s given me a great reputation, albeit not true, as there are other MPs who have done science.
- It’s also a great privilege to represent Cambridge, a town with a great political history. Previous MPs include Oliver Cromwell and Isaac Newton (who was perhaps not a very good MP, but quite a good scientist).
- I used to be a scientist, working in the Cavendish Laboratory. I know some of you are from the more technical side – I also used to write Perl.
- Now I’ve ended up in parliament, which is a very, very strange place indeed.
- Its very old-fashioned; if I ask parliamentary questions, they are then printed out and hand-delivered. Some of the MPs work well within that role. There are some that are quite scary in their attitude to science (I would even define it as anti-science).
- But there are also some that are pro-science, and not just those qualified in science. People who get it. The vast majority – the sort who found science scary at school, don’t really get it, feel there are too many arguments about it – tend to run away from scientific issues, not because they don’t like them, but because its a bit scary.
- That is the challenge – and they can get engaged if you approach them in the right way.
- It’s about dealing with a fundamental way that politicians work and are judged. If you ask someone a question to which they don’t know the answer, they will attempt to answer it. If you ask a politician, the worst thing they can say is say I don’t know. For example, randomized control trials mean that necessarily you’re doing the wrong thing for some people. The same principle applies to government support for startup companies (“Why invest in ones that fail, why not just invest in the ones that will succeed?”).
- The other problem in politics is that if you change your mind on something – you try an experiment and it doesn’t work – it’s not called learning from experience, it’s called u-turning. We expect politicians to never ever change their minds, and that’s a real problem. This government has been criticized for changing its mind on things. We should be allowed to – not on values, but on details.
- I was involved in the debate on the Defamation Bill, and changed my mind. Someone told me that it was the first time someone in the chamber had admitted they were wrong!
- We should be able to have ideas, test them, and then decide. For example, Andrew Lansley’s health reforms could have been a pilot, then we would have known whether they were good or not, but politicians are allergic to doing a pilot.
- It’s not just MPs who are to blame – civil servants are too. For example: We need to protect children, so you shouldn’t have explosives in school, which became “no exothermic reaction” – the person in charge didn’t understand the implications. There is a shortage of people recruited with scientific backgrounds.
- I have ideas about how the government should use science and evidence – my policy paper will be presented at the next LibDem conference. It deals with money (investments in science); putting more money into applied research; science teachers in schools; funding for grad students; and immigration.
- What do we need to do? Make science count. Adjust your vote based on how people think about the things they are talking about. If voters start caring, more MPs will start to think about it. Get engaged – come and talk to your MP.
- I’m a climate scientist working at British Antarctic Survey, and also working 1-2 days a week as a science liaison between the research community and the civil service.
- Ben Goldacre once said that he wanted a t-shirt reading: “It’s more complicated than that”; the relationship between science and technology, and policy-making is definitely “more complicated than that”.
- My thoughts on this have evolved over the last couple of years. I support Mark’s cause, but some aspects i would say are “more complicated than that”.
- For example: the idea, quite common among some, that if only the rest of the world thought like scientists it would be a better place. I’ve come to realize that some scientists don’t always think in that logical way. Take some matter outside your professional area – you still have an opinion, but not necessarily based on logical, scientific and rigorous thinking. That’s just human nature. Part of engaging with people is understanding the way they are thinking. Mutual understanding is critical in developing the relationship between the science community and policy makers.
- Regarding the assertion that it would be better if more scientists were MPs; I was discussing climate science with an MP who has a scientific background, who came up with the classic statement that he found it difficult to be convinced about the case for human-based climate change because CO2 is such a small proportion of the gas in the atmosphere. It’s not clear that having better scientific training automatically leads to better scientific approach to policy making.
- Thirdly, the idea that there is an answer in different policy frameworks (e.g phonics) – I think there are many areas where there isn’t an answer.
- Input from the “geek” community can be helpful by showing how the scientific/tech community deals with uncertainty. Making policy decisions in the face of uncertainty effectively would be beneficial work. One example: raising the height of the Thames Barrier. That decision is made in the face of uncertain scientific information; we don’t know how sea levels will change in the future.
- A sophisticated decision-making process is needed to feed the policy-making process. It’s more complex than it seems, but there is an opportunity to further embed ‘geekness’ into policy decision-making… but it needs to be done through mutual understanding.
- The key thing that I’ve got out of working in Whitehall is a bridge – I’m able to explain to people in Whitehall what is possible, what can be done, what evidence scientists can provide; and I can also go back to the scientific community to explain what is desired by policy makers. There may be scientists with data or information that could be crucial, but is not being presented in the right way.
- Bringing it together, what we need to focus on as a geek community is to recognise and value different parts of the ‘tree’ of evidence-based science. Basic research (the ‘roots’) which might not always have a clear outcome; at the top of the tree (the ‘leaves’) are the applied researchers or private sector companies who handle the interface between what is being done and what is required. We also need the links (the ‘trunk’) between applied research and fundamental research. It’s our responsibility to make sure that tree is there.
Mark – what have you learned since you wrote The Geek Manifesto?
I want to answer some of the comments from Emily. Of course there is a difference between how I portray the issues in a ten minute talk, and how they are presented in the book. Yes, it’s true that not all scientists think scientifically and logically all the time or that every viewpoint they hold is formed through rational argument. The difference is that people with scientific background are more aware of that fact [human nature] and try to counter it – that is what distinguishes a scientific viewpoint. And of course, not all MPs should be scientists (that would be as bad as if they were all lawyers) – being a scientist doesn’t make you a good politician. But having a few more to contribute in the informal/semi-informal discussions that take place, especially as you mentioned uncertainty, would be a good thing. What would I have done differently? I would have done more historical and international background. Time constraints.
Julian, want to respond?
Yes, I agree that it is “more complicated than that”. We won’t have a perfect world – everyone being a scientist wouldn’t be a perfect world. For me, fear is the issue. It’s about having people who get it, who can engage, who can ask. MPs don’t have that many conversations with people in the real world about policy issues. It’s very rare for anyone to want to talk to me about anything that is coming up.
After a short break there was some further Q&A, and closing comments from each speaker. The overall message was one of participation – get involved, find out what is happening within your community, and speak to your MP.