Call for papers: QJEP Special Issue on megastudies, crowdsourcing and big datasets in psycholinguistics

A QJEP Special Issue on megastudies, crowdsourcing and big datasets in psycholinguistics will be edited by Emmanuel Keuleers (Ghent University) and Dave Balota (Washington University, St. Louis)

We invite papers for a special issue of the Quarterly Journal of Experimental Psychology on recent advances in megastudies and crowdsourcing methods and on the use of large non-experimental data sources. The issue will address both the collection of data and the use of these data to answer important theoretical questions.


In recent years, methods of data collection in psycholinguistics have been rapidly evolving along several dimensions.

First, there is a trend towards establishing large laboratory-based experiments without constrained research questions. In these megastudies, behavioral measures are collected for many items using tasks such as lexical decision and naming or sentence reading, with forerunners such as the English Lexicon Project (Balota et al., 2007) or the Dundee Corpus (Kennedy & Pynte, 2005). The number of available datasets produced using the megastudy approach is rapidly increasing for different languages and using different experimental paradigms (Hutchison et al., in press, Cohen-Shikora & Balota, 2013; Keuleers et al., 2010.).

Another recent trend is to gather behavioral data using crowdsourcing rather than laboratory methods (Mason & Suri, 2011). New norms for variables such as Age-of-Acquisition are being successfully collected using Amazon Mechanical Turk (e.g., Kuperman, Stadthagen-Gonzalez & Brysbaert, 2012), and large scale word-association studies are quickly gaining momentum (e.g., De Deyne, Navarro & Storms, 2012). Recent research in Belgium and the Netherlands shows that it is even possible to recruit hundreds of thousands of participants to participate in a lexical decision experiment ( The use of smartphone technology appears also may revolutionize data collection in these large scale studies, as exemplified the Dufau et al. (2011) study of mega lexical decision study of 7 different languages.

In addition to the controlled data collection methods described above, psychologists have been increasingly using freely generated behavioral data, such as text corpora, to extract behaviorally relevant measures. With the increased availability of text sources, particularly subtitles from film and television, high quality word frequency norms are becoming available for various languages.

An exciting trend in this regard is that researchers have been using these text sources to operationalize existing psychological constructs traditionally collected using subjective evaluation (e.g., Bestgen & Vincze, 2012) or to extend learning theory to large-scale learning models (Baayen et al., 2011)

Examples of topics for this special issue:

  1. Papers addressing important theoretical issues using rigorous analyses of megastudy or crowdsourcing data. Preferably, these articles should address the same issue using multiple data sources and use state of the art statistical and computational methods. Articles that use data collection beyond their intended purpose are especially welcomed.

  2. Papers addressing methodological issues with the collection of large datasets, either introducing new methodology or critically evaluating current methods.

  3. Papers presenting new data collected using megastudy or crowdsourcing methods or presenting new measures derived from large corpora.

We aim for a body of high quality articles that introduces and encourages the collection and analysis of large datasets to a large audience and encourages the use of novel data sources and new data collection methods in the research community.

Time Line

September 22, 2013 (or shortly after): Send initial proposals, abstracts of max 400 words to

January 23, 2014: Submission of manuscripts

March 23, 2014: Initial round of reviews

May 23, 2014: Second round of reviews

Fall 2014: Publication

References Baayen, R. H., Milin, P., Djurdjević, D. F., Hendrix, P., & Marelli, M. (2011). An amorphous model for morphological processing in visual comprehension based on naive discriminative learning. Psychological review, 118(3), 438.

Balota, D. A., Yap, M. J., Hutchison, K. A., Cortese, M. J., Kessler, B., Loftis, B., … Treiman, R. (2007). The English lexicon project. Behavior Research Methods, 39(3), 445–459.

Bestgen, Y., & Vincze, N. (2012). Checking and bootstrapping lexical norms by means of word similarity indexes. Behavior Research Methods, 44(4), 998–1006. doi:10.3758/s13428-012-0195-z

De Deyne, S., Navarro, D. J., & Storms, G. (2012). Better explanations of lexical and semantic cognition using networks derived from continued rather than single-word associations. Behavior research methods, 1–19.

Dufau, S., Duñabeitia, J.A., Moret-Tatay, C., McGonigal, A., Peeters, D., Alario, F.-X., Balota, D.A., Brysbaert, M., Carreiras, M., Ferrand, L., Ktori, M., Perea, M., Rastle, K., Sasburg, O., Yap, M.J., Ziegler, J.C., & Grainger, J. (2011). Smart phone, smart science: How the use of smartphones can revolutionize research in cognitive science. PLoS ONE, 6, e24974

Hutchison, K. A., Balota, D. A., Neely, J. H., Cortese, M. J., Cohen-Shikora, E. R., Tse, C.-S., … Buchanan, E. (2013). The semantic priming project. Behavior Research Methods. doi:10.3758/s13428-012-0304-z

Kennedy, A., & Pynte, J. (2005). Parafoveal-on-foveal effects in normal reading. Vision research, 45(2), 153–168.

Keuleers, E., Diependaele, K., & Brysbaert, M. (2010). Practice Effects in Large-Scale Visual Word Recognition Studies: A Lexical Decision Study on 14,000 Dutch Mono- and Disyllabic Words and Nonwords. Frontiers in Psychology, 1. doi:10.3389/fpsyg.2010.00174

Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M. (2012). Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods. doi:10.3758/s13428-012-0210-4

Mason, W., & Suri, S. (2011). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23. doi:10.3758/s13428-011-0124-6

Comments are closed.