I intentionally did not address what statistics is/does, nor what role it should have in the cognitive (neuro)sciences. Many of the attitudes and misunderstandings are all too familiar to me and are topics for a different blogpost (series). In the Twitter debate that spawned those comments, there is also a legitimate concern about the mismatch between the level of statistical expertise required and the level present in the cognitive sciences.

Instead, as indicated in the tl;dr, I really wanted to address the sociological aspect of insulting a group and then patronizing them instead of actually apologizing. But what bothered me even more is that the vast majority of my test readers warned me that I was burning bridges / poking the bear / creating all sorts of problems for myself. This is horrible. A senior, male figure with a tenured – i.e. secure – position at an American R1 institution (i.e. a person of great privilege) is intentionally ignorant, insulting and patronizing and calling that out is viewed as risky? Wow.

Dear all,

Recently, the community entered a fury of discussion based on a few comments made by prominent members of the community. Some of these tweets really bothered me and pointed to profound ignorance about the role of statistics and lack of sympathy for statisticians. Let me address a few of these tweets:

Doing statistics should be like going to the bathroom. Yes, you have to do it. Yes, when you do it, you want to do it right. But don’t make a big deal out of it, be careful about telling other people how to do it, and if your whole life is centered on it, there’s something wrong.

Statistics is not going to the bathroom.

I am appalled that a senior scientist takes pride in willful ignorance and then insults an entire community of researchers before “apologizing” by claiming to speak for that community (of which I am a member):

And dividing and conquering, bringing statisticians onto our papers in author roles raises problems. There aren’t enough statisticians to serve as authors and reviewers on all our papers. And it’s limiting for the statisticians themselves, as they spend less time in lead roles.

In my experience, it goes the other way: methods people are willing to take on such roles, but are unable to find academic jobs and thus are forced into the private sector.

Or as a different senior scientist told me a few years ago, “methods people don’t get [academic] jobs” – such prophecies are by their nature self-fulfilling.

I suspect part of the reason psychology and psycholinguistics are so willing to accept statistical incompetence and malpractice is that it doesn’t matter if the statistics are wrong. Nobody will die. But countless careers and young minds will be wasted chasing noise, as the replication crisis has shown us.

Or to use the original toilet humor: your (statistical) hygiene practices in your private (data) space can have big implications for the health of the broader community. As we all learned in 2020, it’s important to listen to experts and follow best practices.

Phillip Alday

Statistical Nihilism is a somewhat joking belief that there are no true effects. A charitable interpretation is a form of skepticism, heavily informed by my professional experiences and an increasing understanding of the myriad issues underlying statistics as commonly practiced (low power, poor data designs, inappropriate methods and hidden multiple testing). It’s mixed with a bit of disillusionment and disenchantment about what this means for science, both in academia and in industry.

I’m planning on getting back into blogging and two major topic areas are my departure from my previous career trajectory and common statistical/technical/inferential issues I see in my research fields. That might make the mixture of snark and seriousness in statistical nihilism more apparent. But also might be strangely appropriate if there’s no effect of best laid plans.

The very first sentence of the abstract from Marr and Poggio (1976) is:

The CNS needs to be understood at four nearly independent level of description: (1) that a twhich the nature of a computation is expressed; (2) that at which the algorithms that implement a computation are characterized; (3) that at which an algorithm is committed to particular mechanisms; and (4) that at which the mechanisms are realized in hardware.¹

So, it’s four levels, not three, and it’s Marr and Poggio. Yes, I understand that it’s common today to collapse levels (2) and (3) together into one level, but that’s not what Marr and Poggio wrote. They clearly viewed them as distinct levels.

This is why it’s so important to actually read the original articles and not just keep doing indirect quotations.

Parametric statistics are not appropriate for mutual information because the values are non-normally distributed.

Source: Cohen, Michael X. (2014). Analyzing Neural Time Series Data. MIT Press. p 405.

That’s not quite true. Not all parametric statistics depend on normality; indeed the whole class of generalized linear models removes the normality assumption and replaces it with another, appropriate, but still parametric distributional assumption.

Parametric does not mean “assumption of normality” nor does non parametric mean “assumption free”. Parametric simply means that a given distribution (and hence statistical model) can be described via parameters to a given function or functions (in particular, the probability density function, PDF, and the cumulative density function, CDF¹). For example, the normal distribution is described by the probability density function

Setting the parameters (population mean) and (standard deviation) (or equivalently , the variance) completely determines the distribution, giving it a nice, easy-to-compute closed form. Nearly every distribution that you can name is ‘parametric’ in this sense: Student’s , Pareto, Gamma, Cauchy, Laplace, Beta, Binomial, etc. So it’s quite possible that there is a parametric statistical model that fits a given problem, even when conditional normality² can be completely excluded. In any case, attempts to ‘correct’ a statistical model based on the normality assumption when normality is known not to hold are likely to have other latent issues and tradeoffs.

As an alternative to parametric statistics and their distributional assumptions (i.e. that the conditional data distribution can be expressed as a given parametric distribution), non parametric statistics are often presented as an ‘assumption free’ if computationally more expensive alternatives. Computer time is cheap nowadays and the loss of power for modern non parametric methods compared to parametric methods is minimal even when the latter’s assumptions hold, so it seems natural to just use these. (Indeed, Rand Wilcox’s promotion of robust – which goes beyond ‘non parametric’ – statistics is based on this combined with an observation going back to Tukey that most bell-curves are not normal and so the normality assumption may often be invalid.) That’s fine, but these methods also have lots of assumptions, some of them quite strong. For example, the bootstrap assumes that the samples are independent and identically distributed (i.i.d.). In the case of mutual information applied to different EEG channels on the same participant, no two simultaneous measurements from a single person’s scalp can be said to be truly independent (Gauss’ law, spherical conductors, and all that jazz), so this assumption is violated. Violating testing assumptions isn’t the end of the world, but it does mean that you may experience what C programmers dread, namely, undefined behavior.

Now, I do suspect that a permutation test is probably the best option for comparing (within the significance-testing framework) these information-theoretic measures in EEG work. But that’s not because the data aren’t normally distributed, but rather we have no idea what the underlying “ground-truth” distribution is!

You need to remember that “likelihood” is a technical term. The likelihood of , , and the posterior probability of , , are different quantities and they can have different values. The likelihood of is the probability that confers on , not the probability that confers on . Suppose you hear a noise coming from the attic of your house. You consider the hypothesis that there are gremlins up there bowling. The likelihood of this hypothesis is very high, since if there are gremlins bowling in the attic, there probably will be noise. But surely you don’t think that the noise makes it very probable that there are gremlins up there bowling. In this example, is high and is low. The gremlin hypothesis has a high likelihood (in the technical sense) but a low probability.

Source:

Sober, E. (2008). Evidence and Evolution: the Logic Behind the Science. Cambridge University Press.

As quoted in this answer to this question on CrossValidated.

I’m guessing this is related to the name structure used. Project pages are http://username.github.io/repository and User pages are http://username.github.io, so Project pages are in a broad sense “subdirectories” of the User pages. Initializing the User page thus blocks the existing subdirectories, in case you have a subdirectory in the User repository with that same name. Changing the Project gh-pages branch re-initializes / refreshes the Project page, restoring the mount point.

What happens when you have a name collision?

```{r}
?mtcars
```

does not work, because the online help works by side effect. Using help(...,help_type) instead of ? doesn’t help either.

StackOverflow as usual provides a place to get several useful answers. I’ve adapted / extended this answer to display the online help for a given data set inside a box.

<div style="border: 2px solid black; padding: 5px">
```{r, echo=FALSE, results='asis'}
tools:::Rd2HTML(utils:::.getHelpFile(help(mtcars)),stylesheet="")
```
</div>

This will also work for any thing accessible via the online help, but is less ideal for longer help pages because it placed completely inline. For longer help pages, you should probably add a maximum height and scrollbar to the outer div. Also, I’m not sure that the resulting document is 100% well-formed, because the quick-and-easy presented here doesn’t strip the <html>, <head> and <body> tags from the document being embedded.

The output looks like this (note that CSS styles from the embedding document are carried over):

require(graphics)
pairs(mtcars, main = "mtcars data")
coplot(mpg ~ disp | as.factor(cyl), data = mtcars,
       panel = panel.smooth, rows = 1)

You can drastically speed up R in many cases by using a better/more tuned BLAS (linear algebra library) implementation. ATLAS (Automatically Tuned Linear Algebra System) is one such option, but the basic compiled version distributed with Debian and Ubuntu won’t help you out much — you get the most benefit when you compile it yourself so that it’s optimally tuned for your system. (This is also why Debian stopped distributing semi-optimized builds as binary packages.)

No worries though, installing ATLAS from source isn’t hard, and Debian and Ubuntu even maintain a source package.

On Ubuntu and Debian, installing ATLAS goes likes this:

user@localhost:~$ sudo apt-get build-dep atlas
user@localhost:~$ sudo apt-get install build-essential dpkg-dev cdbs devscripts \ 
 gfortran liblapack-dev liblapack-pic 
user@localhost:~$ sudo apt-get source atlas
user@localhost:~$ cd atlas*
user@localhost:~$ sudo fakeroot debian/rules custom

If you get a warning about cpufreq not being set to performance mode, then you’ll have to change that for each CPU (numbered from 0). This will turn off frequency-scaling, which offers a speed boost in and of itself, but perhaps isn’t the best option for laptops as it increases power consumption.

Set n equal to the number of cpus (including virtual HT cpus) minus one:

user@localhost:~$ for i in {0..n}; do sudo cpufreq-set -g performance -c$i; done 

After you set your CPU to performance mode, you can try again:

user@localhost:~$ sudo fakeroot debian/rules custom
user@localhost:~$ sudo apt-get install libatlas-base-dev libatlas-base
user@localhost:~$ sudo dpkg -i libatlas3gf-*.deb 

I recommend using tab completion instead of wildcards, but it should now be installed. You can switch back and forth between different BLAS implementations (and see which one is active) with:

user@localhost:~$ sudo update-alternatives --config libblas.so.3

There are 2 choices for the alternative libblas.so.3 (providing /usr/lib/libblas.so.3).
Selection    Path                                    Priority   Status
------------------------------------------------------------
* 0            /usr/lib/atlas-base/atlas/libblas.so.3   35        auto mode
  1            /usr/lib/atlas-base/atlas/libblas.so.3   35        manual mode
  2            /usr/lib/libblas/libblas.so.3            10        manual mode

Press enter to keep the current choice[*], or type selection number: 

I was happy with auto-selection favoring ATLAS, so I just hit enter.

Now for the benchmarks using large matrix multiplication in R.

Before:

> a = matrix(rnorm(5000*5000), 5000, 5000) 
> b = matrix(rnorm(5000*5000), 5000, 5000) 
> system.time( a%*%b )
   user  system elapsed 
191.668   0.108 191.828 

After:

> a = matrix(rnorm(5000*5000), 5000, 5000) 
> b = matrix(rnorm(5000*5000), 5000, 5000) 
> system.time(a%*%b)
   user  system elapsed 
 34.726   0.200  17.687

That’s more than a 10x speedup in elapsed time!

Sources

• http://packages.debian.org/unstable/libatlas3-base
• http://anonscm.debian.org/viewvc/debian-science/packages/atlas/tags/3.8.4-2/README.Debian?revision=38541&view=markup
• http://wiki.debian.org/DebianScience/LinearAlgebraLibraries
• https://gist.github.com/palday/5685150