Please use this identifier to cite or link to this item: https://doi.org/10.1109/ICDE.2011.5767907
Title: Towards exploratory hypothesis testing and analysis
Authors: Liu, G. 
Feng, M.
Wang, Y. 
Wong, L. 
Ng, S.-K.
Mah, T.L.
Lee, E.J.D.
Issue Date: 2011
Publisher: IEEE
Source: Liu, G.,Feng, M.,Wang, Y.,Wong, L.,Ng, S.-K.,Mah, T.L.,Lee, E.J.D. (2011). Towards exploratory hypothesis testing and analysis. Proceedings - International Conference on Data Engineering : 745-756. ScholarBank@NUS Repository. https://doi.org/10.1109/ICDE.2011.5767907
Abstract: Hypothesis testing is a well-established tool for scientific discovery. Conventional hypothesis testing is carried out in a hypothesis-driven manner. A scientist must first formulate a hypothesis based on his/her knowledge and experience, and then devise a variety of experiments to test it. Given the rapid growth of data, it has become virtually impossible for a person to manually inspect all the data to find all the interesting hypotheses for testing. In this paper, we propose and develop a data-driven system for automatic hypothesis testing and analysis. We define a hypothesis as a comparison between two or more sub-populations. We find sub-populations for comparison using frequent pattern mining techniques and then pair them up for statistical testing. We also generate additional information for further analysis of the hypotheses that are deemed significant. We conducted a set of experiments to show the efficiency of the proposed algorithms, and the usefulness of the generated hypotheses. The results show that our system can help users (1) identify significant hypotheses; (2) isolate the reasons behind significant hypotheses; and (3) find confounding factors that form Simpson's Paradoxes with discovered significant hypotheses. © 2011 IEEE.
Source Title: Proceedings - International Conference on Data Engineering
URI: http://scholarbank.nus.edu.sg/handle/10635/40128
ISBN: 9781424489589
ISSN: 10844627
DOI: 10.1109/ICDE.2011.5767907
Appears in Collections:Staff Publications

Show full item record
Files in This Item:
File Description SizeFormatAccess SettingsVersion 
2011-towards_exploratory_hypothesis_testing_analysis-postprint.pdf216.7 kBAdobe PDF

OPEN

Post-printView/Download

SCOPUSTM   
Citations

6
checked on Dec 13, 2017

Page view(s)

106
checked on Dec 9, 2017

Download(s)

118
checked on Dec 9, 2017

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.