--- title: "Getting Started with smartcor" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Getting Started with smartcor} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` ## Why smartcor? Researchers compute correlations all the time, but the right method depends on the types of the two variables. Most software defaults to Pearson, which suits two continuous variables with a linear relationship. For binary, ordinal, or categorical variables, other methods (point-biserial, phi, polychoric, polyserial, Cramer's V, Kendall's tau, and others) are better suited. Published research shows how often this is ignored. For the accompanying paper (Harshvardhan and Ranjan, 2026), we traced correlation usage through the replication supplements of published economics articles: across 197 papers issuing at least one correlation command, 92.3% of the calls were Pearson, and 88% of the papers used Pearson and nothing else, whatever the variable types involved. We also recomputed 3,322 variable pairs from those supplements. The type-appropriate method differs from the authors' choice in 8.1% of the pairs with no ordinal variable, and in all 732 pairs that involve an ordinal variable. Where the methods differ, the median change in the coefficient is small ($|\Delta r| \approx 0.017$), but it goes as high as $0.47$, and in 36 pairs the change moves the result across $p < 0.05$. **smartcor** does this for you. It detects the variable types, runs a few diagnostics, picks a method, and tells you why it picked it. ## Installation smartcor is available on [CRAN](https://CRAN.R-project.org/package=smartcor): ```{r install, eval = FALSE} install.packages("smartcor") ``` A Python implementation, [pysmartcor](https://pypi.org/project/pysmartcor/), is available on PyPI (`pip install pysmartcor`) and is kept at feature parity with the R package. ## Quick Start ```{r library} library(smartcor) ``` ### Pairwise Correlation The main function is `smart_cor()`. You pass it two vectors: ```{r pairwise} # continuous + continuous: Pearson unless there are outliers or nonlinearity smart_cor(mtcars$mpg, mtcars$wt) ``` The rationale for this pair says that Mardia's test rejects bivariate normality. Pearson is kept because the relationship is linear and free of outliers, but its interval and p-value are approximate. ```{r pairwise-binary} # continuous + binary: biserial when the diagnostics support latent normality smart_cor(mtcars$mpg, mtcars$vs) ``` ## Diagnostics Before choosing a method, `smart_cor()` runs a few checks on the pair. Which ones run depends on the variable types, and each is also available as a function. `detect_outliers()` applies the outer-fence rule: values below $Q_1 - 3\,\mathrm{IQR}$ or above $Q_3 + 3\,\mathrm{IQR}$ are flagged. ```{r outliers} detect_outliers(mtcars$mpg) detect_outliers(c(mtcars$mpg, 120)) ``` `test_bivariate_normality()` picks the normality test from the types. Two continuous variables get Mardia's test. A continuous variable paired with a binary or ordinal one gets the Jarque-Bera test plus the likelihood-ratio ($G^2$) test of the bivariate normal threshold model. Two ordinal or binary variables get the $G^2$ test alone. ```{r normality} test_bivariate_normality(mtcars$mpg, mtcars$wt) test_bivariate_normality(mtcars$mpg, mtcars$vs) test_bivariate_normality(mtcars$gear, mtcars$carb) ``` For continuous/count pairs, `test_linearity()` regresses each variable on the other and orders the residuals by the regressor. A direction rejects linearity when a residual structure test (Box-Pierce, Ljung-Box, or BDS) and RESET both reject, and Spearman is selected only when both directions reject. ```{r linearity} diagnostics = test_linearity(mtcars$mpg, mtcars$wt) diagnostics ``` The result does not depend on argument order. Inconclusive diagnostics retain Pearson and report the reason. Non-rejection does not prove linearity. Every test uses the significance level `alpha`, which defaults to 0.05 and is an argument of `smart_cor()`, `smart_cormat()`, `smart_cor_df()`, and `compare_methods()`. The tests that ran are stored in the `diagnostics` element of the result; entries that did not run are `NULL`. ```{r diagnostics-element} res = smart_cor(mtcars$mpg, mtcars$vs, alpha = 0.01, verbose = FALSE) names(res$diagnostics) res$diagnostics$normality ``` An explicit `method` skips all diagnostics. ### Ordinal Variables When both variables have a small number of unique values, smartcor detects them as ordinal: ```{r ordinal} # numeric variables with 10 or fewer unique values are treated as ordinal; # assume_latent_normal controls whether polychoric or Kendall is used smart_cor(mtcars$gear, mtcars$carb, assume_latent_normal = FALSE) ``` ### Categorical Variables ```{r categorical} # some categorical data set.seed(1) colour = factor(sample(c("red", "blue", "green"), 100, replace = TRUE)) shape = factor(sample(c("circle", "square", "triangle"), 100, replace = TRUE)) smart_cor(colour, shape) ``` A continuous variable paired with a categorical one gets the correlation ratio $\eta^2$, or its Kruskal-Wallis version when the within-group residuals have outliers or are far from normal: ```{r continuous-categorical} smart_cor(mtcars$mpg, factor(mtcars$cyl)) ``` ### Override Type Detection If the detected type is wrong, you can set it yourself: ```{r override} # force cyl to be treated as continuous smart_cor(mtcars$mpg, mtcars$cyl, y_type = "continuous") ``` ### Force a Specific Method ```{r force-method} smart_cor(mtcars$mpg, mtcars$wt, method = "spearman") ``` ## Correlation Matrix `smart_cormat()` computes all pairwise correlations for a data frame, selecting a method for each pair: ```{r cormat} mat = smart_cormat( mtcars[, c("mpg", "cyl", "vs", "gear")], assume_latent_normal = FALSE ) mat ``` ## Tidy Output Both `smart_cor()` and `smart_cormat()` support `tidy()`, which returns a data frame that works with tidyverse tools. The last four columns summarise the diagnostics (`outlier_flag`, `normality_p`, `latent_p`, `linearity_p`) and are `NA` when a test did not run. ```{r tidy} res = smart_cor(mtcars$mpg, mtcars$wt, verbose = FALSE) tidy(res) ``` ```{r tidy-mat} tidy(mat) ``` ## Available Methods Use `available_methods()` to see what methods are available for a given variable-type pair: ```{r available} available_methods("ordinal", "ordinal") available_methods("continuous", "binary") available_methods("categorical", "categorical") ``` ## Method Selection Logic The method selection follows the decision matrix of Harshvardhan and Ranjan (2026), drawn from simulation studies comparing correlation methods: | Variable types | Default method | Alternative | |:--------------------------|:----------------------------------------|:------------------------------------------------------------------| | *Correlation methods (signed, from -1 to +1)* | | | | Continuous + continuous | Pearson (linear, no outliers) | Spearman (outliers or nonlinear), Kendall's tau | | Continuous + binary | Biserial (if latent normality holds) | Point-biserial; rank-biserial (outliers or non-normal) | | Binary + binary | Pearson = phi | Tetrachoric (latent normal), Yule's Q (no latent continuum) | | Continuous + ordinal | Polyserial (if latent normality holds) | Kendall's tau ($m \le 5$) or Spearman | | Ordinal + ordinal | Polychoric (if latent normality holds) | Kendall's tau ($m \le 5$) or Spearman; Goodman-Kruskal gamma | | Binary + ordinal | Polychoric (if latent normality holds) | Rank-biserial; Kendall's tau, Spearman | | *Association measures (unsigned, from 0 to 1)* | | | | Continuous + categorical | Correlation ratio $\eta^2$ | Kruskal-Wallis $\eta^2_H$ (outliers or non-normal); Cramer's V (binned) | | Binary + categorical | Cramer's V | -- | | Ordinal + categorical | Freeman's theta | Cramer's V | | Categorical + categorical | Cramer's V | Theil's U, Tschuprow's T | $m$ is the number of categories of the ordinal variable (the smaller of the two for an ordinal pair). Count variables are treated as numeric continuous variables, so each count combination follows the corresponding continuous row. The `vignette("theory")` vignette explains each rule, and the decision framework article on the package website shows one example per pair. The simulation studies in the paper found that: 1. **Pearson = Point-Biserial = Phi** for 0/1-coded binary variables (exact mathematical equivalence). 2. **Polychoric/polyserial** are unbiased when the latent normality assumption holds, but can be biased when it is violated. 3. **Kendall's tau** is the most robust measure across different latent distributions, which is why smartcor falls back to it (or to Spearman) when latent normality is rejected. 4. **More ordinal categories** = less attenuation, with the biggest improvement from 2 to 3 categories. ## The Latent-Normality Decision For ordinal and binary variables, and for a continuous variable paired with one, the default `assume_latent_normal = "auto"` runs the likelihood-ratio $G^2$ test of `test_bivariate_normality()` and follows its result: ```{r auto} # runs the LR test and picks polychoric or Kendall accordingly smart_cor(mtcars$gear, mtcars$carb, verbose = FALSE)$rationale ``` A binary-binary pair is the exception: its 2 x 2 table is saturated (df = 0), the test cannot run, so smartcor uses phi. `assume_latent_normal = TRUE` gives tetrachoric and `FALSE` gives Yule's Q. For the other pairs, `TRUE` or `FALSE` skips the $G^2$ test and fixes the answer, but the outlier and Jarque-Bera checks still run. Set the argument to `NULL` to be asked interactively (non-interactive sessions then default to `TRUE`). ## Dependencies The latent-variable methods rely on **polycor** (polychoric, polyserial, biserial, and tetrachoric estimates, with their standard errors). Mardia's test comes from **MVN** and the Jarque-Bera test from **moments**. All three are listed under `Imports`, so they install with smartcor. A latent-variable estimator can fail numerically on some data, for example with a singular Hessian or a sparse contingency table. smartcor does not stop with an error when this happens. For the polyserial correlation fitted with `polyserial_ml = TRUE` it first retries the fit from a safe starting value, since polycor's default start for the correlation can land outside $(-1, 1)$ on strongly correlated pairs and abort the optimiser before it takes a step; this fixes most failures. Only if no fit can be obtained does smartcor fall back to Kendall's tau (for polychoric), Spearman (for polyserial), or point-biserial (for biserial); the returned `method`, `method_label`, and `rationale` then name the method actually used, so you can always see which method was used. ## Optional matrix adjustment `nearPD = FALSE` is the default. Enable it explicitly when a downstream procedure requires a positive-definite matrix: ```{r nearpd} adjusted = smart_cormat(mtcars[, c("mpg", "wt", "disp")], nearPD = TRUE, verbose = FALSE) adjusted$nearPD ``` `adjusted$correlations` contains the adjusted matrix; `adjusted$original_correlations` retains the original. Methods, details, and `tidy()` describe the original pairwise estimates and inference. Resolve missing or infinite entries before adjustment. `smart_cor_df(..., nearPD = TRUE)` returns the adjusted correlation table and original method table, without the extra diagnostics.