STEP 3: Study implementation

Study implementation refers to the development of a task or paradigm that will be used to manipulate the independent variables (IVs) and measure the dependent variables (DVs) of interest, as well as the creation of the necessary stimuli and control conditions. Here, stimulus control refers to the methods used to control and manipulate the stimuli that participants are exposed to during the experiment. This might include creating specific visual, auditory, tactile, or other types of stimuli, as well as controlling the timing, duration, and intensity of the stimuli. The selection of openly available stimuli on platforms such as the ➜ Kapodi Stimuli database or ➜ International Affective Picture System is recommended enhancing not only reproducibility, but also ensuring the use of stimuli that underwent a proper standardization procedure (Lang et al., 2008). ARIADNE helps in providing curated, tried-and-tested resources. Crucially, and of note, the trap of “questionable measurement practices” as indicated by Flake (2020) should be avoided by favoring materials proven for standardization, reliability, and validity (e.g., stimuli, tasks, questionnaires; ➜ APA PsycTests or ➜ Open Test Archive). However, researchers should consider that task reliability can mean different things in experimental and correlational research (Hedge et al., 2018; Nebe et al., 2023). Other aspects of study implementation may include the development of a standard operating procedure (SOP; Maghani, 2011; see Table 1) or protocol to guide the experimenter through the study, the creation of a data collection and analysis plan, and the implementation of procedures to ensure the reliability and validity of the study (see also Step 6). It is immensely helpful to note down decisions and the reasons for these decisions, as those will be relevant for the later writing process (Step 8). In this context, preregistration, which entails documenting and uploading the research plan before the outset of data collection, including the hypothesis, design, and analysis plan, has received a great deal of attention recently and been employed as a crucial tool in transparent and reproducible scientific research (Toth et al., 2021; see Table 1; ➜ PROSPERO for systematic reviews, ➜ Open Science Framework templates, or ➜ PreReg). This practice helps to prevent an inflation of the false-positive rate by reducing researcher degrees of freedom and/or limiting decisions within the garden of forking paths (see Table 1). Furthermore, it improves transparency and reproducibility of the study (Peikert et al., 2021). An extension of preregistration, so-called Registered Reports (Henderson & Chambers, 2022; see Table 1), even shift the peer-review process from after to before data collection, allowing researchers to get feedback on their work early in the process and to be able to adapt their research design before the study starts (Scheel et al., 2021).