Abigail Morrison: Communicating Ideas in Computational Neuroscience
Part of the Advances and Challenges in Computational Biology, hosted by PLoS Computational Biology
In computational neuroscience, the key ideas to be communicated are mathematical and computational models as well as data analysis methods. She will mainly focus on computational models in this talk, though what she says mostly holds for data analysis methods as well. They type of modelling being done is getting more and more complicated all of the time. And yet, there is no standardization in notation, simulation software, and best practices for describing models. As a result, we cannot reproduce the work of others or critically evaluate or compare models.
A researcher comes up with an interesting model and simulation, and then they want to publish it. So they try to write down what they did in the model. Then another researcher in a similar area reads it and wants to reproduce or build on it. Then, they run into problems: how do they figure out what parameters to use, what dynamics are present? Ultimately, the system they’re running their simulations on will probably be different, and their version of the model won’t work right. So, she’s working on a system that can be more standardized. Abigail Morrison can think of only one model, in all the times she’s worked on it, that they’ve been able to reproduce without going back to the authors.
Is it science, or is it travel reporting?
Approaches to solve this problem have to be both sociological (large collaborations with defined software and protocols) and technological (version control, high-level APIs, testing and unit testing), or even socio-technological (work together to create tools to facilitate reproducibility).
A lot of interesting work is happening with INCF and NeuralEnsemble/PyNN. INCF has been running since 2005 and tries to coordinate neuroinformatics (databases & data sharing, tool development and analysis, computational models) internationally. INCF also involved with portals, standards incl. ontologies. The Japanese node focuses on the visual side of things, and has produced Visiome, which attempts to collect both papers and figures separately as well as model parameters, simulation scripts and figure-generation scripts. This can all be downloaded, and then hopefully run it on your own system. Another project there is the Simulation Server Platform, intended to provide online test trials for simulation scripts on a virtual machine. All elements are reproduced in the VM such as OS and hardware emulation, compilers, simulation software and viewers. In this way, it supports reproducibility of results by other researchers and testing by journal reviewers.
At the German node, the main focus is to support interactions between experimental and computational neuroscientists, and focuses on collaboratively develop Open-source tools for data access and analysis. The problem is that there are many different recording devices and analysis tools, and no standardization. So, they want a unified data format, implement open source input and export functions for common data formats. Develop and provide a repo for these tools. They also want to design and implement a machine-readable declarative language to describe neural network model (like SBML) – first meeting in March 2009 so still new.
NeuralEnsemble provides hosting for open-source Python-based software projects in Neuroscience, and a key project is PyNN, a common scripting language for all simulators. This facilitates cross-checking of results between simulators, and incremental porting of a model from one simulator to another.
There’s a paper coming in PLoS later this year making a checklist of common suggestions for how network models could be described in words.
Allyson’s thoughts: what about standardization efforts for format/syntax/scope? CARMEN? MIBBI-like efforts: is the checklist effort part of MIBBI? Also, is it really “reproducibility” of results if you have to go to a VM somewhere to get it to work? Probably not, but at least it’s a first step on the road to better tyes of (more complete/generic) reproducibility.
Please note that this post is merely my notes on the presentation. They are not guaranteed to be correct, and unless explicitly stated are not my opinions. They do not reflect the opinions of my employers. Any errors you can happily assume to be mine and no-one else’s. I’m happy to correct any errors you may spot – just let me know!