Cigarette smoke is a complex mixture of more than 8000 smoke constituents. The quantification of selected mainstream smoke constituent yields is one of the methods to evaluating and comparing the performance of different products. Numerous regulatory and scientific advisory bodies have used cigarette smoke constituent yield data for reporting and product comparison purposes. For more than a decade limitations of the indiscriminate application of traditional statistical methods such as the t-test for differences in comparative smoke constituent yield assessments lacking a specific study design, have been highlighted. In the present study, the variability of smoke constituent yields is demonstrated with data obtained under the ISO smoking regime for the Kentucky reference cigarette 3R4F and one commercial brand, analyzed on several occasions between 2007 and 2014. Specifically it is shown that statistically significant differences in the yields of selected smoke constituents do not readily translate to differences between products, and that tolerances need to be defined. To this end, two approaches have been proposed in the literature - minimal detectable differences, and the statistical equivalence. It is illustrated how both approaches provide more meaningful comparison outcomes than the statistical t-test for differences. The present study provides considerations relevant for comparative tobacco product assessments both in the scientific and regulatory contexts.