Viruscraft evaluation and ask-me-anything

Viruscraft is a game about how viruses jump between host species, and co-evolve with their hosts - it’s something we began developing a few years ago, that recently became a bit too relevant for our liking. The project started as a collaboration with evolutionary biologist Dr. Ben Longdon at the University of Exeter, funded by the Wellcome Trust. The result of that was a screen-based game world with evolving host species, together with a tangible interface in the form of a wood virus where the player could manually change the ‘surface protein’ shapes to evolve their virus, with the aim of infecting host species and keeping their virus alive as long as possible. This was used as an installation at the Eden Project and the University of Exeter’s Science in the Square festival.

One thing we particularly liked about it was that while the kids were playing the game, their parents often used the chance to ask us and Ben about things like vaccination and antibiotic resistance. It was popular with young children (who were drawn to the plug in shapes), competitive teenagers, and adult gamers.

We’ve been very aware that although we *thought* the game seemed to be working well for people, we hadn’t had the chance to do any proper evaluation on what ideas people were taking from it. We had a niggling issue with how players had to choose how to evolve their virus depending on what was happening in the host population, and whether that might make people think that evolution is a bit more directed than it is in reality. As we’re now working to develop Viruscraft further with another evolutionary biologist called Ben, Dr. Ben Ashby at the University of Bath, we wanted to make sure that the evaluation happened before making the second version of the game.

Thanks to some Wellcome Trust/University of Exeter public engagement funding, we got a bit of time to work on the evaluation with both Bens, and psychologist/cultural evolution researcher Dr. Lotty Brand. Of course all our brilliant plans were scuppered by a certain real virus, and we ended up redesigning the process to run online - not ideal, particularly since the game was designed in exhibition form. So, as part of the process we did a little work cleaning up an online-only version of the game.

We ran an online survey with the game embedded within it, so people would answer a few questions before playing, then play for ~10 mins, then answer the same questions again after.

This all went though the University of Exeter’s ethical approval system to make sure it was deemed a reasonable approach (this is a major benefit of working with academic institutions), which resulted in a limitation of only allowing people >16 years old taking part (a bit of a shame given that <16s are probably the biggest demographic for the game). It’s also now quite normal within academic surveys to pay participants in some way, either through micropayments (e.g. through MTurk, which brings with it ethical questions of using the Amazon platform) or through a prize draw. Lotty arranged for a £50 prize draw for participants, which could go to charity if the winner wanted. We also arranged an Ask-Me-Anything session with the two Bens, which was open to all the participants.

Online surveys have a bit of a problem in that they can end up going to a narrow demographic, for example if we only advertised via academic twitter feeds, most respondents would be academics. To broaden this out a bit, we printed postcards inviting people to take part which went through doors in Falmouth Old Hill (an area that has ranked in the top 10% on the Index of Multiple Deprivation), advertised on local Covid groups, and through our normal social media channels. The survey was also completed by a small group of Further Education students as part of their studies.

From 42 participants, aged 16 to over 80, we found that the game improved people’s understanding of how viruses infect hosts. The game didn’t significantly improve or degrade people’s understanding of virus mutations, or how viruses jump between hosts. This is mixed news - we're definitely achieving one of the core aims (how viruses infect hosts), but not the other (how viruses jump between hosts). Understanding the nature of virus mutations (random as opposed to directed) wasn't an original aim, but we were concerned that people might be being left with the wrong idea about how mutations happen, so it was reassuring not to find any strong evidence of this. The results from Lotty are in full at the end of this post (including some interesting findings on players' confidence levels).

Ask-me-anything sessions

We ended up splitting the Ask-Me-Anything session into two, with one for the Further Education students and one for others, both were facilitated and in both cases people could either send in questions in advance or ask them during the session. We had 23 people take part in total. The questions people asked are useful for understanding what they are interested in and what people struggle with, some of which could inform the next game version. We see huge scope for this simple format, as incredibly low-hanging fruit for connecting local researchers directly with their broader communities. It would be a particularly easy thing for universities to run (and complementary to opening up their seminars to anyone, which has always seemed like an obvious and easy step to take).

We don’t have a transcript of the Q&A for the student group, but these were their questions:

• How do you get to be a virus person?!
• How do you develop the first vaccine for a virus?
• How were viruses discovered?
• How do viruses survive?
• Why are viruses so good at mutating?
• There are good as well as bad Bacteria, are there any beneficial viruses?
• How good would you say our country's response to COVID was?
• Why are viruses so alien?
• How does the covid vaccine work for different strains?
• How long do you reckon it'll take for a vaccine resistant strain of COVID to appear?
• What is a viroid?
• How long does it take before a vaccine is effective in the anatomy?
• Are new viruses mutating all the time?
• How are there different strains of covid?

We do have a rough transcript of the Q&A for the other group:

Q: What are the things sticking out of the virus that you change in the game?

They are the surface proteins of the virus. Like when you see a picture of a virus on the news and it looks like a ball with spikes sticking out of it, the things you change in the game are the spikes. Followed by an explanation of the ‘lock and key’ mechanism.

Q: How does the virus ‘know’ how to mutate its surface shapes?

It doesn’t… mutations are happening randomly all the time as the virus replicates. The replication process isn’t perfect so mistakes creep in, and these are mutations. It’s a bit like photocopying something over and over again, it looks less like the original over time. Sometimes those mutations are good for the virus (e.g. it means they can infect a new host), sometimes they are bad for the virus, but most of the time they are neither good or bad. The problem with letting the Covid-19 virus spread in populations like we have in the UK is that the virus is replicating itself all the time, and that means there is a large number of mutations taking place all the time - it’s like we’re constantly rolling the dice, and some of those random mutations are likely to bad (e.g. make the virus better at transmitting).

Q: What work are you currently doing, or would you like to do with stakeholders, like other university departments, governments, NGOs etc.?

Planning for the next pandemic - we need to do better at our preparedness, we knew this was coming but weren’t prepared to deal with it or to understand its impacts, and although the pandemic has been very bad, the next one could be worse.

Q: How long do viruses last on surfaces?

Q follow on: That’s a really good question because I’m currently quarantining things for 3 days and it would be good to know if that’s not enough?

It’s a really hard thing to answer, there are lots of studies that give different answers, and they all need to be treated cautiously because there are so many things that aren’t always taken into consideration. As an example, if I had Covid-19 and sneezed on my screen now, how many virus particles would there be on the screen and how long would they last - well the answer would depend on so many things: How big was my sneeze? How much virus was in my sneeze? What temperature is the air and the screen? How humid is it? How much UV light was there from the sun that day?

Q. What are the two Bens doing to keep themselves safe?

Ben A: We know that it’s mostly airborne now, so while before I was opening gates with sticks or leaves, I’m now less bothered by touching things outside that other people might have touched. If I was clinically vulnerable I would wash my shopping I think. Sometimes I hold my breath when I’m walking past people!

Ben L: Increased levels of handwashing after handling shopping. I was being more careful with things like shopping deliveries, but now only really wipe things that have been refrigerated (e.g. milk that is handled a lot) - there have been a lot of outbreaks in places like meat packing facilities, which are cold, and cold allows the virus to live longer. I do leave post/deliveries on the side for a day or so, or tip it out of the envelope and then wash my hands.

Neither Bens go into supermarkets and both work at home, recognising that they are very fortunate to be able to do that.

Q. Has seasonal flu been eradicated and if not why isn’t it killing people?

Colds and flu have gone to really low levels but have not been eradicated. We’re expecting that there might be a bigger cold and flu season in future years because most people have missed a year of building up immunity. The interesting thing is that this pandemic has shown us that if we wanted to we could substantially reduce seasonal flu deaths, which kill about 10,000 a year in the UK, or up to 20,000 people in a bad year - that should be something that we talk about as a society - would we be willing to have seasonal actions like wearing masks in the winter to reduce these deaths?

Q. What would be the one thing you would want the public to know or do as a result of your research?

Ben L: I’d hope people now have a better understanding of evolution more generally, like understanding that viruses just naturally mutate all the time, and it’s not always a bad thing. We keep seeing headlines that are very dramatic about mutations.

Ben A: I’d want people to understand better that disagreement between scientists happens, but that doesn’t mean that each ‘side’ has equal weight. Some scientists have been pushing for naturally-acquired herd immunity, but the mathematical modelling of the impacts of that hasn’t been done, so we don’t have evidence that it’s a good approach. Most scientists disagree with chasing herd immunity by allowing people to be infected because the cost in terms of deaths would likely be so high - there are lots of vulnerable people out in the community who would be at risk, and vulnerable people in care homes wouldn’t be immune so just one person coming in could be a huge risk to them.

Q. If you had a crystal ball, what will we be talking about in 6 months?

Both Bens are cautiously optimistic about the vaccine rollout, and how things will look in 6 months. It’s very positive that we’ve managed to very quickly produce multiple, effective vaccines, which was by no means a guarantee.

Ben L: You expect a mutation burst when a virus jumps into a new host, which is what’s happened with this pandemic - we expect some adaptive mutations at the moment and that’s exactly what we’re seeing.

Ben A: I hope we’re talking more about the next pandemic, and planning better for it. Places with a joined up health service will be better able to deal with this, and in the UK we are lucky having a very good NHS and good civil servants, which is why the vaccine rollout is going quite well - places like the US will struggle more.

Full survey results

All data, analysis scripts and results can be found at https://github.com/lottybrand/viruscraft

Demographics

Overall we had 42 participants take part who all

1. Were aged over 16
2. Gave full consent to take part
3. Passed the attention check question
4. Answered some/all questions of interest

Out of these 42 participants, 29 had completed some form of Higher Education (either undergraduate or postgraduate degree) and 13 had not.

Out of these 42, 36 lived in the UK and 6 did not. 29 lived in Cornwall, and 13 did not.

The age breakdown of the participants was as follows:

16 to 25 = 6

26 to 34 = 15

35 to 44 = 7

45 to 56 = 10

65 to 80 = 3

over 80 = 1

Confidence

The mean confidence level of participants’ understanding of virus transmission before playing Viruscraft was 3.74 (sd 1.71) (1 = low confidence, 7 = high confidence).

This increased to 3.90 (sd 1.32) after playing.

Interestingly, those without higher education had higher confidence levels (4.25) before playing than those with higher education (3.5), but these decreased after playing the game (3.44), whilst the confidence of those with higher education increased (4.10). In fact, out of the 18 participants with higher education, confidence increased after playing the game for 9 of them, for 6 of them it stayed the same, and it decreased in 3 participants. In contrast, for the 9 participants without higher education, only 1 participant’s confidence increased after playing the game, for 2 it stayed the same, and for 6 it decreased after playing the game.

This decrease in confidence for those without higher education is consistent with previous findings in psychology in which being questioned about the mechanisms behind a complex topic which you believe you understand often reveals gaps in one’s knowledge and decreases one’s confidence compared to your baseline confidence before being questioned (Rozenblit, L., & Keil, F. (2002). The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive science, 26(5), 521-562.)

Virus infection

For a virus to infect a host, what must happen?

1. Its DNA must match the host’s DNA
2. A protein on its surface must match the shape of the host’s receptor
3. It must be transported by bacteria into the host
4. I don’t know

Before playing the game, 25/42 (60%) participants were correct in their answer on virus infection, i.e. that “a protein on the virus surface must match the shape of the host’s cell.” If participants were guessing, we would expect 25% correct given there were four possible responses, two incorrect answers and one “I don’t know” option. 8 participants selected “I don’t know” and 9 selected the wrong answer “The virus must be transported by bacteria into the host.”

After playing the game, 28/32 (88%) answered this question correctly, with only one selecting “I don’t know,” and 3 selecting the wrong answer relating to bacteria.

When statistically modelling these data, we found strong evidence to suggest that playing the game increased the proportion of correct answers (95% confidence intervals: 0.44, 2.06).

Virus mutation

Which of the following statements are true?

a) Mutations are always negative

b) Mutations are unnatural

c) Mutations happen randomly

d) I don’t know

Before playing the game, 33/42 (79%) participants were correct in their answer on virus mutation, i.e. that “mutations happen mostly randomly.” If participants were guessing, we would expect 25% correct given there were four possible responses, two incorrect answers and one “I don’t know” option. 6 participants selected “I don’t know” and 3 selected the wrong answer “mutations are always bad for the host.”

After playing the game, 23/32 (72%) answered this question correctly, with only 3 selecting “I don’t know,” but 6 selecting the wrong answer that mutations are always bad for the host.

When statistically modelling these data, we did not find strong evidence to suggest that playing the game altered the proportion of correct answers (95% confidence intervals: -1.06, 0.77).

Virus Evolution

Sometimes viruses are able to jump from one species to another. How does this happen?

a) If a new species is near to the virus, the virus will automatically infect it

b) Viruses constantly change their protein shapes to try to fit new species

c) When viruses reproduce, some randomly have different proteins, which match new species’ receptors.

d) I don’t know

Before playing the game, 19/42 (45%) participants were correct in their answer on virus evolution, i.e. that “when viruses reproduce, some randomly have different proteins which might match a new host species’ cells.” If participants were guessing, we would expect 25% correct given there were four possible responses, two incorrect answers and one “I don’t know” option. 8 participants selected “I don’t know”, 13 selected the wrong answer “viruses constantly change their protein shapes to try to fit new host species,” and 2 selected the wrong answer “if a new host species comes into contact with the virus, the virus will always be able to infect it.”

After playing the game, 17/32 (53%) answered this question correctly, with only 3 selecting “I don’t know,” 9 selecting the wrong answer relating to viruses constantly changing shape, and 3 selecting the wrong answer relating to species coming into contact with the virus.

When statistically modelling these data, we did not find strong evidence to suggest that playing the game altered the proportion of correct answers (95% confidence intervals: -0.57, 1.08).