STEP 3: Study implementation#
In contrast to Step 2, which focuses on planning the research project, Step 3, Study Implementation, involves translating those plans into action. It entails developing tasks or paradigms to manipulate independent variables (IVs) and measure dependent variables (DVs), as well as creating necessary stimuli and control conditions. While Step 2 sets the theoretical and methodological framework, Study Implementation (Step 3) addresses the practical execution, including the precise crafting of stimuli and control over experimental conditions. This phase ensures the smooth transition from theory to practice, focusing on the creation of tasks, stimuli, and control conditions to achieve the research objectives outlined in the study design.
Stimuli selection and standardization. The selection of openly available stimuli on platforms such as the ➜ Kapodi Stimuli database or ➜ International Affective Picture System (IAPS) is recommended enhancing not only reproducibility, but also ensuring the use of stimuli that underwent a proper standardization procedure (Lang et al., 2008). Crucially, and of note, the trap of “questionable measurement practices” as indicated by Flake & Fried (2020) should be avoided by favoring materials which have been tested for standardization, reliability, and validity (e.g., stimuli, tasks, questionnaires; ➜ APA PsycTests or ➜ Open Test Archive). However, researchers should consider that task reliability can mean different things in experimental and correlational research (Hedge et al., 2018; Nebe et al., 2023).
Developing procedures and protocols. Other aspects of study implementation may include the development of a standard operating procedure (SOP; Maghani, 2011; see glossary) or protocol to guide the experimenter through the study, the creation of a data collection and analysis plan, and the implementation of procedures to ensure the reliability and validity of the study (see also Step 6). It is immensely helpful to note down decisions and the reasons for these decisions, as those will be relevant for the later writing process (Step 8). For example, the decision for a suitable task programming environment should take into account whether the study will be lab-based or implemented online and whether the program is available at no costs (➜ Psychopy vs. ➜ Psychtoolbox in Matlab).
Preregistration and Registered Reports. In this context, preregistration, which entails documenting and uploading the research plan before the outset of data collection, including the hypothesis, design, and analysis plan, can be employed as a crucial tool in transparent and reproducible scientific research (Toth et al., 2021; see glossary; ➜ PROSPERO for systematic reviews, ➜ Open Science Framework templates, or ➜ PreReg). This practice helps to prevent an inflation of the false-positive rate (see Type I error in Table 1) by reducing researcher degrees of freedom and/or limiting decisions within the garden of forking paths (see glossary). Furthermore, it improves transparency and reproducibility of the study (Peikert et al., 2021). An extension of preregistration, so-called Registered Reports (Henderson & Chambers, 2022; see glossary), even shifts the peer-review process from after to before data collection, allowing researchers to get feedback on their work early in the process and to be able to adapt their research design before the study starts (Scheel et al., 2021).
Key questions in this step#
What measures are most fitting (tasks, questionnaires, etc.)?
What stimuli need to be created (e.g., pictures, videos, text)?
Which programming environment should be used?