File Download
There are no files associated with this item.
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1080/01621459.2023.2218030
- Find via
Supplementary
-
Citations:
- Appears in Collections:
Article: Asymptotic Distribution-Free Independence Test for High Dimension Data
Title | Asymptotic Distribution-Free Independence Test for High Dimension Data |
---|---|
Authors | |
Issue Date | 26-May-2023 |
Publisher | Taylor and Francis Group |
Citation | Journal of the American Statistical Association, 2023 How to Cite? |
Abstract | Test of independence is of fundamental importance in modern data analysis, with broad applications in variable selection, graphical models, and causal inference. When the data is high dimensional and the potential dependence signal is sparse, independence testing becomes very challenging without distributional or structural assumptions. In this paper, we propose a general framework for independence testing by first fitting a classifier that distinguishes the joint and product distributions, and then testing the significance of the fitted classifier. This framework allows us to borrow the strength of the most advanced classification algorithms developed from the modern machine learning community, making it applicable to high dimensional, complex data. By combining a sample split and a fixed permutation, our test statistic has a universal, fixed Gaussian null distribution that is independent of the underlying data distribution. Extensive simulations demonstrate the advantages of the newly proposed test compared with existing methods. We further apply the new test to a single cell data set to test the independence between two types of single cell sequencing measurements, whose high dimensionality and sparsity make existing methods hard to apply. |
Persistent Identifier | http://hdl.handle.net/10722/331012 |
ISSN | 2023 Impact Factor: 3.0 2023 SCImago Journal Rankings: 3.922 |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Cai, Zhanrui | - |
dc.contributor.author | Lei, Jing | - |
dc.contributor.author | Roeder, Kathryn | - |
dc.date.accessioned | 2023-09-21T06:51:58Z | - |
dc.date.available | 2023-09-21T06:51:58Z | - |
dc.date.issued | 2023-05-26 | - |
dc.identifier.citation | Journal of the American Statistical Association, 2023 | - |
dc.identifier.issn | 0162-1459 | - |
dc.identifier.uri | http://hdl.handle.net/10722/331012 | - |
dc.description.abstract | <p>Test of independence is of fundamental importance in modern data analysis, with broad applications in variable selection, graphical models, and causal inference. When the data is high dimensional and the potential dependence signal is sparse, independence testing becomes very challenging without distributional or structural assumptions. In this paper, we propose a general framework for independence testing by first fitting a classifier that distinguishes the joint and product distributions, and then testing the significance of the fitted classifier. This framework allows us to borrow the strength of the most advanced classification algorithms developed from the modern machine learning community, making it applicable to high dimensional, complex data. By combining a sample split and a fixed permutation, our test statistic has a universal, fixed Gaussian null distribution that is independent of the underlying data distribution. Extensive simulations demonstrate the advantages of the newly proposed test compared with existing methods. We further apply the new test to a single cell data set to test the independence between two types of single cell sequencing measurements, whose high dimensionality and sparsity make existing methods hard to apply.<br></p> | - |
dc.language | eng | - |
dc.publisher | Taylor and Francis Group | - |
dc.relation.ispartof | Journal of the American Statistical Association | - |
dc.title | Asymptotic Distribution-Free Independence Test for High Dimension Data | - |
dc.type | Article | - |
dc.identifier.doi | 10.1080/01621459.2023.2218030 | - |
dc.identifier.eissn | 1537-274X | - |
dc.identifier.issnl | 0162-1459 | - |