Introduction to the statistical inference tool
 Student’s ttest
 MannWhitney U test
 Paired student’s ttest
 Wilcoxon signedrank test
 KruskalWallis H test
 Oneway analysis of variance (Oneway ANOVA)
 Twoway analysis of variance (Twoway ANOVA)
How to run a significance test
A workflow for setting up a statistical significance test
Introduction to the statistical inference tool
The statistical inference tool in the Cytobank platform offers several statistical hypothesis testing methods that are widely used by researchers to analyze biological data. You can directly run a significance test in the Cytobank platform using results from manual gating or machine learningbased analysis, such as FlowSOM, without exporting analysis results manually outside of the Cytobank platform.
The following is a list of statistical inference methods implemented in the Cytobank platform:
Figure 1: Statistical inference methods implemented in the Cytobank platform.
 Significance tests:
 Multiple testing and correction methods:

 Tukey’s HSD
 Bonferroni correction
 FDR correction – Benjamini & Hochberg procedure

For the detailed definition of the implemented methods, please see below
Note: Please see the key statistical terms section for the definition of statistical terms and How to set up a paired statistical test for more details on sample pairing.
Significance tests
 Student’s ttest (also known as ttest)
Student’s ttest is a parametric hypothesis test to determine if there are any differences in group means between the two groups of samples of an independent variable. It assumes that the data follows a normal distribution.
Example: You want to determine if Treatment A lowered the percentage of CD4+ T cells that are Tregs as compared to Treatment B. Perform a Student’s ttest to compare the percentage of Treg cells of CD4+ T cells between a group of mice that was treated with Treatment A to a group of mice treated with Treatment B, if you know that the percent of Tregs follows a normal distribution.
MannWhitney U test is a nonparametric hypothesis test used to determine if the distributions of two sample groups of an independent variable are equal or not. It does not assume the data follows a parametric distribution, such as normal distribution.
Example: You want to determine if Treatment A lowered the percentage of CD4+ T cells that are Tregs as compared to Treatment B. Perform a MannWhitney U test to compare the percentage of Treg cells of CD4+ T cells between a group of mice that was treated with Treatment A to a group of mice treated with Treatment B, if you know that the percent of Tregs does not follow a normal distribution.
 Paired student’s ttest
Paired student’s ttest is a parametric hypothesis test to determine if there are significant differences between groups in an experimental design with paired samples, such as if samples are from individuals pre and posttreatment. Like the student’s ttest, paired student’s ttest assumes that the data follows a normal distribution and determines if the means of two groups are equal or not.
Example: You want to determine if a treatment lowered the percentage of CD4+ T cells that are Tregs after treatment as compared to before treatment, in the same mice. Perform a Paired student’s ttest to compare the percentage of Treg cells of CD4+ cells measured on the same mice pre and posttreatment, if you know that the percent of Tregs follows a normal distribution.
 Wilcoxon signedrank test
Wilcoxon signedranks test is a nonparametric test used to determine if the distributions of two sample groups of an independent variable in a paired experimental design are equal or not. Similar to the MannWhitney U test, Wilcoxon signedrank test determines if two groups of samples have the same distribution or not.
Example: You want to determine if a treatment lowered the percentage of CD4+ T cells that are Tregs after treatment as compared to before treatment, in the same mice. Perform a Wilcoxon signedrank test to compare the percentage of Treg cells of CD4+ cells measured on the same mice pre and posttreatment, if you know that the percent of Tregs does not follow a normal distribution.
KruskalWallis test is a nonparametric method that does not assume the data follows a normal distribution. The test is used to compare more than two groups of an independent variable and determine if all the groups have the distribution or not.
Example: A use case of the test is to assess treatment effectiveness. For example, a study has three groups of patients. They receive different levels of treatment and then report outcomes. A KruskalWallis test can determine if the treatment outcome is corrected with the level of treatment.
Oneway ANOVA is a parametric test that can analyze an independent variable with more than two groups. The test determines if there are any differences in group means among the sample groups.
Example: A use case of Oneway ANOVA is to compare three different age groups of samples. The result can tell if the effectiveness of treatment shows any difference between different age groups.
The twoway analysis of variance analysis is a parametric method. It can analyze two independent variables simultaneously, where the two independent variables have at least two groups. In the Cytobank platform, the twoway ANOVA analysis evaluates the effect of two independent variables without evaluating the effect of the interaction of the two independent variables.
Example: A use case of Twoway ANOVA test is to find out if a new drug or treatment works after controlling for the effect of time. For example, a study may contain two groups of samples. One group receives the new treatment, and the other group doesn’t receive any treatments. All samples are measured at three time points, pretreatment, posttreatment Day 1, and posttreatment Day 7. A twoway ANOVA test can determine if there is a difference between the treatment group and the placebo group regarding less of the difference of the time points.
Multiple testing methods
When more than one statistical test is being performed simultaneously, it is advisable to apply a correction, or adjustment, to pvalues. This correction is applied to avoid finding significant results solely because you have done more tests, known as the “multiple testing problem.” For example, in cytometry data analysis, researchers usually describe the immune system in a sample based on multiple cell populations. When comparing samples, it is common that a statistical significance test is repeated for each cell population, known as “multiple testing.” A selection of multiple testing methods is available to handle this scenario.
Note that in the Cytobank platform, each Table dimension, as configured in the layout settings, is handled independently; multiple testing correction is performed within each table dimension. Pvalues are not corrected across table dimensions.
I. Following a twogroup test (Student’s t test, MannWhitney U test, Paired student’s t test, and Wilcoxon signedrank test), Bonferroni and FDR correction methods are available to select from the Multiple test methods menu.
 Bonferroni correction
Bonferroni correction is a common adjustment method to address the multiple testing problem. It applies a correction based on the number of tests performed together.
 FDR correction  Benjamini & Hochberg procedure, a false discovery rate (FDR) correction method
Benjamini & Hochberg procedure is a multiple testing correction method that is designed to control the false discovery rate. Compared to the Bonferroni method, an FDRcontrolling method provides less stringent control of Type I errors.
II. Following the KruskalWallis H test or the Twoway ANOVA test, a combination of a twogroup test and correction method can be selected, to perform pairwise comparisons between all groups, as well as apply a pvalue adjustment. This will identify which specific groups differ significantly from each other in the case of more than two groups. If Multiple testing is set to “None,” the only significance test results will be the overall pvalue indicating significance among the IV(s).
III. Following KruskalWallis H test:
 MannWhitney U test + Bonferroni
Perform all pairwise comparisons using MannWhitney U test and apply a Bonferroni correction.
 MannWhitney U test + FDR
Perform all pairwise comparisons using MannWhitney U test and apply a false discovery rate correction.
IV. Following Twoway ANOVA test:
 Student’s t test + Bonferroni
Perform all pairwise comparisons using Student’s t test and apply a Bonferroni correction.
 Student’s t test + FDR
Perform all pairwise comparisons using Student’s t test and apply a false discovery rate correction.
If a oneway ANOVA is used, the Tukey’s HSD test is available for pairwise comparisons between each group of the independent variable.
 Tukey’s HSD (honestly significant difference) test
Tukey’s HSD test, also known as Tukey’s range test or Tukey’s test, is a singlestep method for correcting the multiple testing problem and conduct pairwise comparisons. It compares all possible pairs of group means to identify which groups of an independent variable differ significantly from each other.
How to run a significance test
The statistical inference tool is available in the Illustrations Editor page. To access the tool, create a new illustration in the Illustration Editor, and set plot type to be Box, Violin, Bar, Line or Summary dot. Select a Significance test method from the dropdown list that is compatible with your experimental design, based on the number of independent variables (I.V.), number of groups, parametric or nonparametric, and paired or unpaired.
Figure 2: To see the list of significant test methods, click Illustrations > New illustration > Plots > Select a summary plot from the plot type dropdown list.
A workflow for setting up a statistical significance test
 Choose which statistic should be compared and which statistical test to use from the plot settings menu
I. Select a summary Plot type from the Plot type dropdown.


 Additional settings related to the appearance of the plot will appear. See Overview of Summary plots generation using the Illustration Editor for more details on configuring the plot.

II. Choose a Statistic from the Statistic dropdown. This statistic will be calculated for each sample selected for inclusion in the illustration, and will be used for significance testing.
III. Pick a significance test from the Significance test dropdown based on the displayed assumptions and constraints of each test shown.


 If a paired test is selected (Paired Student’s t test or Wilcoxon signedrank test), the Cytobank platform automatically extracts pairing information from the sample tags of an experiment. For more information on setting up a paired test, see this article.

IV. If applicable (performing multiple twogroup tests within each plot, or choosing a KruskalWallis H, Oneway or Twoway ANOVA), choose an appropriate Multipletestingmethod method.
2. Configure the illustration layout from the layout menu
I. Assign the variable defining groups of interest as the independent variable (I.V.). The second row of the layout menu, labelled Subgroups, will be used to assign the I.V. by default. (Note: in some layouts, the I.V. may move to the first row of the layout menu.) Choose an annotated Figure dimension as the Subgroup/IV(s) to be compared.
II. Select at least two Sample tags within the I.V. to indicate sample groups
i. If a twogroup test is selected (Student’s ttest, MannWhitney U test, Paired student’s ttest, or Wilcoxon signedrank test), make sure the figure dimension specified as the I.V. has exactly two groups selected. An error message will appear if you have selected less than or more than two sample tags as groups.
ii. If a twoway ANOVA significant test is selected, two Figure dimensions are required, labelled I.V. #1 and I.V. #2 in the first and second rows of the Layout settings menu.
Figure 3: Workflow to set up a significance test
How to interpret the results
The Cytobank statistical inference tool displays the result of the significance test in two ways.
 Pvalues or asterisks displayed in the Illustration
Pvalues or asterisks are shown within and/or adjacent to the Illustration plot. The format of displayed pvalue or symbols depends on the statistical inference test and the selected Show pvalues option is enabled. Significance symbols correspond to pvalues as described in the table below.
 Twogroup tests (Student’s t test, MannWhitney U test, Paired student’s t test and Wilcoxon signedrank test):
 If Show pvalues is set to Symbols or Pvalues, the selected option is displayed on the plot, centered and underlined between the two groups being compared, either for all comparisons, or only significant comparisons, defined by p <0.05.

 When a followup “multiple testing” option is enabled, the pvalues and corresponding symbols displayed on the plot are the adjusted pvalues for all the significance tests.
Figure 4: Boxplots showing significant results following Student’s ttest (left) and Student’s t test with Bonferroni correction (right)
 Oneway ANOVA, KruskalWallis H test:

 If Show pvalues is set to Symbols or Pvalues, the selected option appears atop a group of data on the plot to indicate that there are significant differences between groups in that Xaxis group, if an Xaxis group is selected. If there is no Xaxis group selected, the symbol represents significant differences between groups among all the samples displayed underneath the symbol or pvalue underline.
 If a multiple testing option is enabled, an additional summary result table appears to the right of each plot if there are any statistically significant (p<0.05) pairwise comparisons. This table lists all significant pairwise comparisons and their corresponding significance symbols.
Figure 5: Bar plots showing Oneway analysis of variance (left) and Oneway analysis of variance with Tukey HSD follow up test significant comparisons (right).
 Twoway ANOVA:

 The names of the significant I.V.’s are displayed adjacent to each plot along with their corresponding significance symbol.

 Enabling a multiple testing method does not change the display of significance on the plot, only the Statistical Inference table (see below)
Figure 6: Box plots showing significant variables identified by Twoway analysis of variance test.
For any tests that cannot be completed, due to missing data, or data that have no variation, NaN will be shown if Show p values is set to P values (all) or Symbols (all).
2. Complete testing results in the Statistical Inference table
A statistical inference table will be displayed in the Illustration when a significance test is performed. The table is located below the summary plots in a section called “Statistical Inference.”
Twogroup tests
For twogroup tests (Student’s ttest, MannWhitney U test, Paired student’s t test, Wilcoxonsigned rank test), each row of the table summarizes one comparison. Depending on the selection of the significant testing methods, the following columns will be displayed in the table.
a. Mean A or Mean B
i. Calculated mean of independent variable groups, referred to as A and B, if the aggregate method in the Plots settings is selected as mean (option available for Bar and Line plots named Summary method)
b. Median A or Median B
i. Calculated median of independent variable groups, referred to as A and B, if the summary method in the Plots settings is selected as median (option available for Bar and Line plots)
c. SD A or SD B
i. Standard deviation of independent variable groups A and B
d. “n A” or “n B”
i. Number of data points of independent variable groups A and B
e. pvalue
i. Unadjusted pvalue
ii. For any tests that cannot be completed, due to missing data, or data that have no variation, NaN will be shown.
f. Adjusted pvalue
i. If a multiple testing method is used following a twogroup test, both the pvalue and adjusted pvalue is shown
ii. If a multiple testing method is used following KruskalWallis H, Oneway or Twoway ANOVA test, the only pvalues to appear in this table will be adjusted.
iii. For any tests that cannot be completed, due to missing data, or data that have no variation, NaN will be shown.
g. Significance pvalue
i. Symbol denoting unadjusted pvalue for the comparison of group A and group B
ii. For any tests that cannot be completed, due to missing data, or data that have no variation, NaN will be shown.
h. Adjusted significance pvalue
i. Symbol denoting adjusted pvalue for the comparison of group A and group B
ii. For any tests that cannot be completed, due to missing data, or data that have no variation, NaN will be shown.
KruskalWallis H, Oneway ANOVA, Twoway ANOVA
If Oneway ANOVA, Twoway ANOVA, or KruskalWallis H test is selected, and “multiple testing” option is on, two result tables will be displayed. The result table at the top is labelled with the test name and shows the omnibus results, describing if the variance among all the groups of the I.V.(s) are significantly different. The following columns are displayed in this table:
 Table figure dimension name(s)

 If a Table(s) dimension is enabled, the name of this dimension appears in the leftmost column(s)
 Xaxis figure dimension name

 If an Xaxis figure dimension is selected for a test with 1 I.V., the name of the dimension appears in the next column. For a Twoway ANOVA test, the Xaxis figure dimension will appear in the I.V. column
 Independent variable (I.V.)

 Name of the independent variable(s) used for comparison
 Pvalue

 Overall pvalue for the variable(s) listed in that table row
 For any tests that cannot be completed, due to missing data, or data that have no variation, NaN will be shown.
 Significance pvalue

 Symbol denoting overall pvalue significance
 For any tests that cannot be completed, due to missing data, or data that have no variation, NaN will be shown.
The second table at the bottom of the statistical inference table section is labelled with the name of the selected multiple testing method and shows the followup analysis results. The columns follow the format of the twogroup test tables (above).
For example, if Oneway ANOVA is performed and Tukey HSD test is selected as the followup analysis, the top result table reports the result of the Oneway ANOVA test. The bottom table reports the results of Tukey HSD test.
Key statistical terms
 Paired samples

 Samples that have been measured multiple times in one experiment. A common example of paired samples is an individual measured before and after experimental intervention. In this case, the Individual is used as the “pairing dimension.”
 Nonparametric vs. parametric

 Parametric methods are statistical inference methods that assume sample data follows a probability distribution that can be described using a set of parameters. For example, Student’s ttest is a parametric method because it assumes sample data follows a normal distribution that can be defined by two parameters, mean and variance. Unlike parametric methods, nonparametric methods do not assume sample data follow a parameterized distribution.
 Independent variable (I.V.)

 An I.V. is an experimental factor that defines an assignment of samples. It is a categorical variable that has a fixed number of possible values. For example, in a dataset that has pretreatment and posttreatment samples, the I.V. of the dataset is a condition factor that defines the two groups of samples. An I.V. is selected using annotated Figure dimensions.
 Groups of an independent variable

 Groups are the possible values of an I.V. For example, consider a dataset with measurements taken on 4 timepoints. The I.V. is the timepoint and there are 4 possible values to group the samples. They are Day 1, Day 5, Day 10, and Day 20. These four possible values are the groups of the timepoint independent variable and are annotated as Sample tags.