Can a mouse understand the crypto market?
Neurons in the mouse brain correlate with cryptocurrency price: a cautionary tale
Abstract
Recommendation: posted 27 August 2021, validated 02 September 2021
Martinez Vergara, H. (2021) Can a mouse understand the crypto market?. Peer Community in Neuroscience, 100002. 10.24072/pci.cneuro.100002
Recommendation
Nowadays it is pretty much accepted that in animals with a nervous system, neural activity leads to behaviour. This framework is very useful to ultimately find a satisfying explanation of how and why animals behave, as it implies that there is a causal relationship between neuronal spiking and muscle and gland activity. In order to get closer to this causation, a common approach in neuroscience is to find correlations between behavioural variables and neuronal activity. Dr. Meijer's manuscript "Neurons in the mouse brain correlate with cryptocurrency price: a cautionary tale" [1] serves as a proof of concept that neuroscientists need to be careful about the statistical tests they use when looking for these correlations.
In this work, the author considers two recent datasets containing signals that display slow continuous trends over time: neuronal spiking activity from 40,100 neurons, and Bitcoin and Ethereum prices. When testing for correlations between the activity of individual neurons and the simultaneous fluctuations of cryptocurrency prices, he finds that over two thirds of the neurons correlated significantly, and that classical corrective conservative methods still result in one third of the neurons showing correlation. In order to estimate the true false discovery rate of these type of comparissons, the author tested two statistical methods shown to work for simulated data [2]. He shows that also for this large-scale dataset, both the session permutation and the linear shift method manage to reduce the number of correlated neurons to statistically-acceptable levels. Additionaly, the author goes on to show that it is the slow time constant of the crypto prices that are the root for the initial correlations. This work serves as an example for how mislead scientists can be if proper statistical tests are not applied in order to avoid "nonsense correlations" with neuronal data, and it aims to increase awareness about this problem in the neuroscience community.
This rigorous and yet entertaining work can now be added to the collection of cautionary tales that include a dead salmon understanding human emotions [3] and rat cortical neurons predicting stock market prices [4]. At the very least, it can be a piece of advice for Elon Musk to wait for more evidence before merging two of his new recent interests.
References
[1] Meijer, Guido. (2021). Neurons in the Mouse Brain Correlate with Cryptocurrency Price: A Cautionary Tale. PsyArXiv, ver. 3 peer-reviewed and recommended by Peer Community in Circuit Neuroscience. https://doi.org/10.31234/osf.io/fa4wz.
[2] K. D. Harris. (2020). Nonsense correlations in neuroscience. bioRxiv. 402719. https://doi.org/10.1101/2020.11.29.402719
[3] http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg
[4] T. Marzullo, C. Miller, and D. Kipke. (2016). Stock Market Behavior Predicted by Rat Neurons2. Annals of Improbable Research 12, 401. PDF Link
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article. The authors declared that they comply with the PCI rule of having no financial conflicts of interest in relation to the content of the article.
no declaration
Evaluation round #1
DOI or URL of the preprint: https://doi.org/10.31234/osf.io/fa4wz
Version of the preprint: 1
Author's Reply, 20 Aug 2021
Decision by Hernando Martinez Vergara, posted 09 Jul 2021
Dear Dr. Meijer,
Thanks for submitting your preprint titled "Neurons in the mouse brain correlate with cryptocurrency price:
a cautionary tale" (https://doi.org/10.31234/osf.io/fa4wz) to PCI C.Neuro.
I have received three different reviews of your manuscript. The general view is that the paper is well written, the methodology is properly employed and explained, and the conclusions derived from the work are well stated and supported by the analysis. Using an open-source large-scale state-of-the-art resource to draw attention to 'non-sense correlations' in neuroscience, exemplified using modern finantial instruments, is a great idea. However, in order for this piece to add significant value to the existing literature, I agree with the reviewers that additional works need to be done to address their points:
- "The underlying cause of the results are, however, not explicitly clarified and could be further looked into by testing the hypothesis whether using signals with temporal autocorrelations that have different time constants still maintains significant correlations."
- "with 60 s long time bins, almost never used in neuroscience studies". The preprint would benefit from adding examples about when these time bins are used and which behavioural correlates have such time constants. Additionally, I would be curious to see how the interplay between the time constant and different time bins affects the results.
Looking forward for your resubmission,
Hernando M. Vergara
Reviewed by Kenneth Harris, 06 Jul 2021
Great illustration of nonsense correlations, and glad the methods worked!
Also how often do you get a chance to read a carefully caveatted statement like "mice almost certainly lack the capacity to read and interpret complex financial data" in a scientific paper.
Reviewed by anonymous reviewer 1, 14 Jun 2021
The preprint aims to warn neuroscientists from the use of standard correlation methods, such as the Pearson correlation coefficient, when comparing slowly changing time series such as cryptocurrency prices against binned (with 60 s long time bins, almost never used in neuroscience studies ) neuronal spiking activity, to estimate neuron tuning. Application of this method leads to a high number of false positive (FP) tuned neurons, even when standard multiple comparison correction methods are applied. In contrary, application of more sophisticated techniques like the session permutation and linear shift method (both extensively described in Harris 2020, biorXiv), leads to a reasonable estimate of FP tuned neurons.
I think the notion that the Pearson correlation coefficient should be taken merely as a first pass indication of potential relation between neuronal firing and variable of interest, requiring validation by ad hoc methods like cross-validated statistical models and linear shift, has now been around the field for many years (one out of many examples: Campagner et al. 2016, elife) and recently reviewed in Harris 2020,biorxiv (as mentioned by the author).Therefore, despite being an interesting exercise, I don't see the relevance nor the novelty of this work.
Reviewed by Anirudh Kulkarni, 07 Jul 2021
The study by Meijer presents an example of the erroneous conclusions that can be reached by using correlations without any systematic control analyses. More specifically, one could be led to the false conclusion that the neural activity in the mouse brain correlates with the fluctuations in the Bitcoin and Ethereum price while not using the appropriate control analyses. The study demonstrates the usefulness of two methods developed recently by K. D. Harris(2020) namely the Permutation method and the LinShift method to test the significance of correlations rather than the standard ‘shuffle’ method that is used in literature. This article serves as a good warning when correlating neuronal activity with any variables including behavioural data.
The article is well written. The abstract clearly represents the methodology used and aims of the article. The introduction highlights the problem in current neuroscience data analyses methods by showing how 'nonsense correlations' in data can lead to false conclusions. It sets out to use the proposed methods in literature to see if they were successful in reducing these false errors that creep into the analysis. The article uses the publicly available spiking activity of 40,010 neurons in 58 mice available from the Allen Institute and the Bitcoin and Ethereum prices using the Python library Historic-Crypto. The Pearson correlation coefficient between the single neuron activity and the cryptocurrency prices have been calculated. The methodology is thus sound and replicable.
Corrections: For Figure 1, it would be good to use consistent colours for the different entities over all the panels: i.e. blue represented the firing rates of the neurons in Fig. 1A whereas it represents Bitcoins in Fig. 1B.
The methods described in the results section i.e. the permutation method and the linear shift method could ideally be moved to the methods section to allow a smoother reading of the results.
The statistical test used to compare the different distributions in Figure 2 and to calculate the p-values needs to be stated explicitly in the methods section.
The discussion is well guided and comments on the results obtained. The underlying cause of the results are, however, not explicitly clarified and could be further looked into by testing the hypothesis whether using signals with temporal autocorrelations that have different time constants still maintains significant correlations.
Minor corrections:
Abstract: line 4 ∼40.000 should be ∼40,000
Methods: line 6 40.010 must be 40,010.
