Research transparency and negative results within iGEM

Survey

For this study, we conducted a survey to find out how the iGEM community perceives the reliability of old wikis. We also wanted to find out how they planned to write their own wiki, with regards to transparency and negative results.

Our population for this survey was all 280 teams registered for iGEM in 2015. Since the population is small the sample size needs to be large, in proportion, to get good statistical significance. Even for the rather weak confidence level of 90 % and a margin of error of 10 %, the sample size must be larger than 55 teams. Our goal was to reach this sample size with a good response rate.

We sent a link to the survey by email to 70 teams, chosen at random from all participating teams in 2015. We chose this method over distributing the link on social media to improve randomization of the sample group and to track the response rate. Email addresses were collected from the "About Our Lab" questionnaires on the iGEM website. Invitations to take part in the study were sent on July 29th 2015. Reminders were sent on August 6th and again on August 13th. When the survey closed on August 19th, 44 teams had responded – a response rate of 63 %. This sample size was smaller than we expected, and our confidence level is thus 85 % with a 10 % margin of error.

In each section of the survey, we wanted to answer a number of questions. Sections were answered independently, and it was possible to move backwards in the survey. It was also possible to save the results and continue with the survey at a later date. This survey had three sections, aimed at three areas of inquiry.

How does the iGEM community use old wikis?

How does the iGEM community perceive negative results?

How does the iGEM community treat negative results on their wiki?

The final question in the survey was "Have you discussed with your team before answering the survey?" ​

How does the iGEM community use old wikis?

Biobricks are like pieces of a machine. To understand how a cogwheel, pulley or lever works in a machine it is best to look at examples. Wikis are important to learn how BioBricks can fit together in large systems. When we researched older wikis we were impressed by what some teams had done and eager to build on their accomplishments. Parts and circuits were often described in language that implied that they worked as intended. We wanted to investigate how old wikis are used in the iGEM community. When we started looking for results for parts relevant to our project, they were often hard to find. We started noticing a pattern of mixing ideas with results in a way which made it difficult to assess what the team had actually accomplished. To see if this experience was common among other iGEM teams, we wanted to know if they perceived that ideas were difficult to distinguish from results on wikis. We also wanted to know if they found it easy or difficult to find results at all.

Even when results are present, the data may be lacking. A result may be presented as positive, but since wikis are not reviewed before publication it is good to look at the data before moving forward with a part or circuit. We wanted to know if iGEM teams had been able to find data on the wikis to support the claims. If a team moves forward with a part from a previous project, they may find out later that the part actually never worked as intended. This can be due to difficulty of finding clear results, but it is also possible that some teams leave out negative result and over emphasize positive results. We wanted to investigate what experience iGEM teams have had with using parts or protocols described on old wikis.

How does the iGEM community perceive negative results?

Publication of negative results is a point of discussion in the research community. But in research, ideas are usually not published without results. Thus, ideas or innovations that do not work are not published. In the iGEM community the question is even more complicated since there is no review of wikis before publication. Some ideas are presented without any experimental data to test them. This is not strange given the short timeframe and the sky's-the-limit atmosphere of iGEM, but it makes it difficult for to tell apart working constructs from lofty ideas. It is possible that some results are missing because they are negative. This is troublesome since a compelling idea might then be attempted again and again by teams that do not publish their negative results. In the second part of the survey we wanted to find out how the iGEM community views negative results.

In the survey we defined negative results. This definition was stated on the top of the page of the section: "A negative result is conclusive and supports the null-hypothesis. By conclusive we mean that the result is significant, has been reproduced multiple times and answers a posed question. In other words, a negative result is one that indicates that your experiment does not work as intended."

We wanted to know if the team had discussed the concept of negative results and how they valued them in relation to positive results. We also wanted to understand why some teams may not include negative results. To investigate this we asked if they worry about how judges would perceive negative results. We also asked if they worried about that negative results might make the wiki confusing to read.

How does the iGEM community treat negative results?

We wanted to know if teams were planning to report all, some or none of their negative results on their wikis. If they planned to omit some negative results, we wanted to know why.

Results

Part I: How does the iGEM community use old wikis?

Question 1

“Wikis are a source of information and inspiration available to everyone. We have read wikis from previous years to…”

Figure 1. Answers to survey question 1; “Wikis are a source of information and inspiration available to everyone. We have read wikis from previous years to…”

Question 2

“Judging from the experience in your team, how easy is it to tell apart ideas from results on the wikis?”

Figure 2. Answers to survey question 2; “Judging from the experience in your team, how easy is it to tell apart ideas from results on the wikis?”

Question 3

“Judging from the experience in your team, how easy is it to find results on the wikis you have read?”

Figure 3. Answers to survey question 3; “Judging from the experience in your team, how easy is it to find results on the wikis you have read?”

Question 4

“Judging from the experience in your team, how easy is it to find statistically significant data on the wikis that supports the results?”

Figure 4. Answers to survey question 4; “Judging from the experience in your team, how easy is it to find statistically significant data on the wikis that supports the results?”

Question 5

“Reading wikis can be an important inspiration to find parts and protocols for your project. What is your experience with using previous wikis in this way?”

Figure 5.“Reading wikis can be an important inspiration to find parts and protocols for your project. What is your experience with using previous wikis in this way?”

Summary

The survey shows that most iGEM teams use wikis from previous years for inspiration and to find out if their project ideas have already been explored by other teams. Many teams also use results, protocols and techniques that they find on wikis. Wikis from previous years are clearly an important source of information and inspiration for iGEM teams.

Most teams don’t struggle to find results on wikis but they have difficulties finding significant data to support the results. Many teams also think it is difficult to separate ideas from results. As one iGEM team said: “The teams state what they are doing, but when you try to search for results, they cannot be found.”

About half of the survey participants had also tried to reproduce claims from other teams. A majority of these teams later found out that the previous team did not achieve what they claimed. Out of the teams asked 33% tried and failed to reproduce another team’s results, while only 14% percent succeeded. This indicates that reproducibility of results within iGEM is quite low.

Part II: How does the iGEM community perceive negative results?

Question 6

“Have you ever discussed negative results with your team?”

Figure 6. Answers to survey question 6; “Have you ever discussed negative results with your team?”

Question 7

“If discussed, how are negative results perceived in your team?”

Figure 7. Answers to survey question 7; “If discussed, how are negative results perceived in your team?”

Question 8

“Have you ever been concerned about how your negative results would look to the judges?”

Figure 8. Answers to survey question 8; “ Have you ever been concerned about how your negative results would look to the judges?”

Question 9

“Have you ever been worried that the wiki would look confusing if negative results were included?”

Figure 9. Answers to survey question 9; “ Have you ever been worried that the wiki would look confusing if negative results were included?”

Summary

iGEM teams clearly think that negative results are important and discuss them within their team. However, they also worry about how negative results may be perceived. Many think that their wiki would look confusing if they include negative results. Almost 80% of the teams are also worried about how negative results would look to the iGEM judges. Some teams particularly expressed their concerns about the judging forms; “We are very concerned that judges view negative results as failure, even if important and/or meaningful information is gained. This is largely due to the judging forms which require success to achieve medals”

Part III: How does the iGEM community treat negative results on their wiki?

Question 10

“Are you considering including your negative results in your wiki?”

Figure 10. Answers to survey question 10; “ Are you considering including your negative results in your wiki?”

Question 11

“If you will not include all of your negative results, why?”

Figure 11. Answers to survey question 11; “If you will not include all of your negative results, why?”

Summary

Despite being worried about the judging, more than 90% of the survey participants say they will include negative results on their wiki. However, many will not include all negative results. The most common reasons to exclude negative results was to make the wiki less confusing or because the results were considered unimportant.

One team said; “If they are significant, we would like to include them. However, there might be results that are not relevant for our project, or that we won't be able to find out and explain why are they negative. This will make incomplete and difficult to understand the wiki.” about including negative results on the wiki.

Wiki Evaluation

In the previous chapter, we could show that a vast majority of newly-formed iGEM teams use the wikis of former iGEM teams to get inspired or to check whether their idea has been tried before (See Figure 1). However, about two thirds of this year’s iGEM participants felt that is rather difficult to find clearly conclusive data. In an attempt to make this subjective impression more quantifiable, we evaluated 19 out of 34 of overgraduate teams from the competition in 2013 and 2014. We put particular focus on how clearly information, ideas, experiments and results are presented on each single iGEM team’s wiki. In an attempt to quantify how well data and information is displayed on wikis, we have developed a “Stockholm iGEM wiki pledge”. There we have included criteria with particular importance which are indispensable for data representation and clarity of experimental findings.

Major parts of the “Stockholm iGEM wiki pledge” are:

A clear border between ideas and results

All ideas are linked to hypotheses, every hypothesis is linked to a follow up

An overarching idea can be visionary but it should be broken down into the ideas that will actually be tested in the scope of the project.

Ideas are further defined as testable hypotheses.

Every hypothesis is connected to one or several practical experiments.

Every stated hypothesis is clearly linked to a written follow-up.

A hypothesis is stated for each experiment

All experiments are listed along with their purposes and the hypothesis they are meant to test. Experiments are linked to their results

A result can be positive, negative, inconclusive, unfinished or not started. Results that are inconclusive, unfinished or not started include an explanation for why that was the case.

Every tested hypothesis clearly linked to a written follow-up.

Results are reported equally and thoroughly

Negative results are reported

Negative results are defined as conclusive results that do not support the tested hypothesis.

Results that are not conclusive should be very clearly defined as such. Data from results that are not conclusive can be omitted from the wiki. Observed trends in case of inconclusive results can be reported in the follow up.

Priority is given to conclusive and critical results

When choosing which result to give the most space on the wiki, priority should be given to results that are conclusive and critical, even if this completely disproves a hypothesis that the project is based on.

Results are presented thoroughly

Conclusive results are presented thoroughly enough that it’s possible for future teams to determine if a part, biobrick or protocol has worked as described in the idea part of the wiki.

Borrowed and attributed ideas are declared

Attributions are clearly stated

Attributions are clearly stated in text each time a non-original idea or result is introduced.

A separate page lists detailed attributions and how they were used in the project.

We want to stress at this point, that the “Stockholm iGEM wiki pledge” is a suggestion from our group for clear delivery of research data on iGEM wikis. Of course, there are other ways to evaluate wiki pages. However, based on the survey shown in the previous chapter, the criteria of the “Stockholm wiki pledge” seem to be in line with the perception of many other iGEM teams.

The attempt to make iGEM wikis quantifiable in their quality represents a very difficult task. Therefore, we needed to have a strict evaluation pattern which prevents to strong subjective influence from single evaluators. Hence, four of our team members have been evaluating 5-6 wikis independently from each other. They performed their evaluation in a pre-set lime survey following the our ”Stockholm wiki pledge” outlined above.

The teams from 2013 and 2014 have been chosen as evaluation basis as we think that particularly overgraduate teams should be aware of the requirements that come along with data representation. By evaluating 19 out of 34 iGEM wikis, we get a significance level of 90% and are allowed to draw first conclusions from our own wiki evaluation.

We evaluated up to 5 hypotheses for a single team, each clearly stated in their own wiki page. By reading through the overview of the project and the experimental results, we examined the relevance of the previously stated ideas to the empirical part of the project. If the team stated less than 5 claims in their wiki page, we assessed the smaller number of the hypotheses accordingly. Less than 5 hypotheses were examined for 4 teams, out of the whole sample population.

After evaluating 19 randomly chosen iGEM wikis, we compiled the data and analysed them by standard statistical tools such as Quick Statistics software and graphical representation of the responses’ distribution.

Using this approach, we hope to answer the question whether data representation on wikis is a major issue in the iGEM community. We want to help improve the quality of future wikis by quantifying the clarity and conclusiveness of previous wikis. Furthermore, we want to identify possible starting points for future iGEM teams, to lift the quality of information representation on their wiki pages.

Results

The results obtained by conducting the wiki evaluation let us assess the connection between ideas, hypotheses, and experimental plans.

A majority of the teams from 2013 and 2014 had clearly connected ideas to the experiments, this accounts for almost 70% of the whole randomized wiki population:

Table 1. Distribution of responses to the statement ‘The idea is clearly connected to the experiment’. No answer indicates lack of a clearly stated hypothesis to be assessed.

Answer	Response	Percentage [%]
Yes	66	69.47
No	8	8.42
Uncertain	9	9.47
No answer	12	12.63

The next question regarded the connection between the purpose of the laboratory experiments and their relevance to theoretical ideas in the project plan. Again, the results of this question’s assessment reveal that, most prevalently, the relation was clear.

However, the increased uncertainty of the evaluation suggests; too ambiguous descriptions, awkwardness of language and lack of straightforward explanations. It also suggests insufficiently explained flow of ideas to experiments.

Table 2. Distribution of responses to the statement ‘The purpose of major experiments connected to this idea is clearly stated’. No answer indicates lack of a clearly stated hypothesis to be assessed.

Answer	Response	Percentage [%]
Yes	67	70.53
No	5	5.26
Uncertain	12	12.63
No answer	11	11.58

In the next question, we decided to evaluate the follow-up of ideas and their declared outcome. Here, the proportion of positive answers was slightly smaller than in the previous questions, accounting for 60% of all responses. With a stable proportion of the answers indicating no clear follow-up, we could observe more uncertainty among the assessors in this question. Usually, this was caused by vague descriptions of the connection between the idea and the experimental outcome

Table 3. Distribution of responses to the statement ‘The idea has a follow-up which declares the outcome.” No answer indicates lack of a clearly stated hypothesis to be assessed.

Answer	Response	Percentage [%]
Yes	57	60.00
No	9	9.47
Uncertain	19	20
No answer	10	10.43

Subsequently, we wanted to assess the wikis in terms of negative results. The results at this point were dramatically opposite. Only one of the four examined teams was able to present their negative results. This accounts for 7% of the whole sample of teams from 2013 and 2014.

Figure 12. Distribution of answers to the statement ‘Negative results are presented’. No answer indicates lack of a clearly stated hypothesis to be assessed.

Next, we aimed to assess if results presented on the wikis were conclusive. In this question, the assessors decided that most of the results presented in the wikis were sufficiently conclusive. Interestingly, a relatively high proportion of the assessments were uncertain. This may indicate poor presentation of results or intentional ambiguity.

Figure 13. Distribution of answers to the statement ‘Presented results are sufficiently conclusive’. No answer indicates lack of a clearly stated hypothesis to be assessed.

Eventually, our team wanted to know if the results are conclusive enough to determine if the project worked as described in the idea part of the wiki. Here the distribution of both ‘Yes’ and ‘No’ responses was somewhat alike and differed by only 7 percentage points.

Figure 14. Distribution of answers to the statement ‘Conclusive results are thorough enough that it is possible to determine if it worked as described in the idea part of the wiki’. No answer indicates lack of a clearly stated hypothesis to be assessed.

Lastly, some general assessors’ comments indicated poor layout, navigation and intuitiveness of some wikis. lack of explanation of some figures and formulas. Nevertheless, the main objection from the assessors side was the definitive lack of negative results.

Conclusions

Recommendations for the future:

Teams should:

clearly separate ideas and results

present raw data/statistical analysis

iGEM could:

help with statistics

explain about importance of negative results

Judges should:

reward negative results.