STEP 4: Piloting#
A pilot study, also known as exploratory trial, is a preliminary small-scale study conducted to assess potential problems, duration, and other factors before a full experimental investigation. This is often a reflective and iterative process (Thabane et al., 2010). By setting criteria based on important feasibility objectives and research goals, pilot studies enable researchers to determine the feasibility of a more extensive, time-consuming, and expensive main study and to test whether the operationalisation (Step 2) makes sense (see ARIADNE for resources related to piloting; e.g., ➜ data simulation). Note that certain study types such as literature reviews might not need a pilot study, but instead require piloting in the sense of testing the search criteria and procedures.
Feasibility. It is common to always test a few “pilot” participants with your whole set-up before starting Step 5 (the Data Collection), making sure that participants understand the instructions of the experiment and all procedures work as planned. The main study can then be improved based on the findings of this pilot. Of note, another complementary method for better determining a study’s feasibility is to simulate data, which allows researchers to set up and test the analysis pipeline, and prepare for prospective outcomes before carrying out the primary investigation.
Validation. Second, on an operationalisation level, these preliminary data should be used to check whether all variables of interest can be extracted from the raw files. Note here that data from pilot studies or participants should ideally be kept separate from the data of the main study as they might differ in the way they were measured. Although preliminary data from pilot studies can be used to estimate effect sizes for sample size calculations of the main study (Sakaluk, 2016), these estimates might be inaccurate because of the small-scale nature of the pilot (Albers & Lakens, 2018). In the worst case, the sample size calculation is based on overestimated effect sizes, leading to the initiation of an underpowered main study that is susceptible to false positive (Type I error) or false negative (Type II error) findings. Sample size calculations from pilot studies should, therefore, be treated with caution (Lakens, 2022).
In conclusion, piloting can be useful for study planning and design, enabling researchers to evaluate viability, foster greater transparency, and enhance the overall quality of their research.
Key questions in this step#
Is the study feasible?*
Do all manipulations work as intended?
Questions with an asterisk indicate that these should ideally already be explored before starting a project.