CRC Press - Computational Methods of Feature Selection - 2008: CRC Press - Computational Methods of Feature Selection - 2008.pdf

CRC Press - Computational Methods of Feature Selection - 2008

Chapman & Hall/CRC
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487‑2742
© 2008 by Taylor & Francis Group, LLC
Chapman & Hall/CRC is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed in the United States of America on acid‑free paper
10 9 8 7 6 5 4 3 2 1
International Standard Book Number‑13: 978‑1‑58488‑878‑9 (Hardcover)
This book contains information obtained from authentic and highly regarded sources. Reprinted
material is quoted with permission, and sources are indicated. A wide variety of references are
listed. Reasonable efforts have been made to publish reliable data and information, but the author
and the publisher cannot assume responsibility for the validity of all materials or for the consequences
of their use.
No part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any
electronic, mechanical, or other means, now known or hereafter invented, including photocopying,
microfilming, and recording, or in any information storage or retrieval system, without written
permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.
copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC)
222 Rosewood Drive, Danvers, MA 01923, 978‑750‑8400. CCC is a not‑for‑profit organization that
provides licenses and registration for a variety of users. For organizations that have been granted a
photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Library of Congress Cataloging‑in‑Publication Data
Liu, Huan, 1958‑
Computational methods of feature selection / authors/editors, Huan Liu and
Hiroshi Motoda.
p. cm. ‑‑ (Chapman & Hall/CRC data mining and knowledge
discovery)
Includes bibliographical references and index.
ISBN 978‑1‑58488‑878‑9 (alk. paper)
1. Database management. 2. Data mining. 3. Machine learning. I. Motoda,
Hiroshi. II. Title. III. Series.
QA76.9.D3L5652 2007
005.74‑‑dc22 2007027465
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com

Preface
It has been ten years since we published our first two books on feature selection
in 1998. In the past decade, we witnessed a great expansion of feature
selection research in multiple dimensions. We experienced the fast data evolution
in which extremely high-dimensional data, such as high-throughput data
of bioinformatics and Web/text data, became increasingly common. They
stretch the capabilities of conventional data processing techniques, pose new
challenges, and stimulate accelerated development of feature selection research
in two major ways. One trend is to improve and expand the existing techniques
to meet the new challenges. The other is to develop brand new algorithms
directly targeting the arising challenges. In this process, we observe
many feature-selection-centered activities, such as one well-received competition,
two well-attended tutorials at top conferences, and two multi-disciplinary
workshops, as well as a special development section in a recent issue of IEEE
Intelligent Systems, to name a few.
This collection bridges the widening gap between existing texts and the
rapid developments in the field, by presenting recent research works from various
disciplines. It features excellent survey work, practical guides, exciting
new directions, and comprehensive tutorials from leading experts. The book
also presents easy-to-understand illustrations, state-of-the-art methodologies,
and algorithms, along with real-world case studies ranging from text classification,
to Web mining, to bioinformatics where high-dimensional data are
pervasive. Some vague ideas suggested in our earlier book have been developed
into mature areas with solid achievements, along with progress that
could not have been imagined ten years ago. With the steady and speedy
development of feature selection research, we sincerely hope that this book
presents distinctive and representative achievements; serves as a convenient
point for graduate students, practitioners, and researchers to further the research
and application of feature selection; and sparks a new phase of feature
selection research. We are truly optimistic about the impact of feature selection
on massive, high-dimensional data and processing in the near future, and
we have no doubt that in another ten years, when we look back, we will be
humbled by the newfound power of feature selection, and by its indelible contributions
to machine learning, data mining, and many real-world challenges.
Huan Liu and Hiroshi Motoda
© 2008 by Taylor & Francis Group, LLC
Acknowledgments
The inception of this book project was during SDM 2006’s feature selection
workshop. Randi Cohen, an editor of Chapman and Hall/CRC Press,
eloquently convinced one of us that it was a time for a new book on feature
selection. Since then, she closely worked with us to make the process easier
and smoother and allowed us to stay focused. With Randi’s kind and expert
support, we were able to adhere to the planned schedule when facing unexpected
difficulties. We truly appreciate her generous support throughout the
project.
This book is a natural extension of the two successful feature selection
workshops held at SDM 20051 and SDM 2006.2 The success would not be
a reality without the leadership of two workshop co-organizers (Robert Stine
of Wharton School and Leonard Auslender of SAS); the meticulous work of
the proceedings chair (Lei Yu of Binghamton University); and the altruistic
efforts of PC members, authors, and contributors. We take this opportunity
to thank all who helped to advance the frontier of feature selection research.
The authors, contributors, and reviewers of this book played an instrumental
role in this project. Given the limited space of this book, we could
not include all quality works. Reviewers’ detailed comments and constructive
suggestions significantly helped improve the book’s consistency in content,
format, comprehensibility, and presentation. We thank the authors who patiently
and timely accommodated our (sometimes many) requests.
We would also like to express our deep gratitude for the gracious help we
received from our colleagues and students, including Zheng Zhao, Lei Tang,
Quan Nguyen, Payam Refaeilzadeh, and Shankara B. Subramanya of Arizona
State University; Kozo Ohara of Osaka University; and William Nace and
Kenneth Gorreta of AFOSR/AOARD, Air Force Research Laboratory.
Last but not least, we thank our families for their love and support. We
are grateful and happy that we can now spend more time with our families.
Huan Liu and Hiroshi Motoda
1The 2005 proceedings are at http://enpub.eas.asu.edu/workshop/.
2The 2006 proceedings are at http://enpub.eas.asu.edu/workshop/2006/.

Contents
I Introduction and Background 1
1 Less IsMore 3
Huan Liu and Hiroshi Motoda
1.1 Background and Basics . . . . . . . . . . . . . . . . . . . . . 4
1.2 Supervised, Unsupervised, and Semi-Supervised Feature Selection
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Key Contributions and Organization of the Book . . . . . . . 10
1.3.1 Part I - Introduction and Background . . . . . . . . . 10
1.3.2 Part II - Extending Feature Selection . . . . . . . . . 11
1.3.3 Part III - Weighting and Local Methods . . . . . . . . 12
1.3.4 Part IV - Text Classification and Clustering . . . . . . 13
1.3.5 Part V - Feature Selection in Bioinformatics . . . . . . 14
1.4 Looking Ahead . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 Unsupervised Feature Selection 19
Jennifer G. Dy
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.1 The K-Means Algorithm . . . . . . . . . . . . . . . . 21
2.2.2 FiniteMixture Clustering . . . . . . . . . . . . . . . . 22
2.3 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1 Feature Search . . . . . . . . . . . . . . . . . . . . . . 23
2.3.2 Feature Evaluation . . . . . . . . . . . . . . . . . . . . 24
2.4 Feature Selection for Unlabeled Data . . . . . . . . . . . . . 25
2.4.1 FilterMethods . . . . . . . . . . . . . . . . . . . . . . 26
2.4.2 WrapperMethods . . . . . . . . . . . . . . . . . . . . 27
2.5 Local Approaches . . . . . . . . . . . . . . . . . . . . . . . . 32
2.5.1 Subspace Clustering . . . . . . . . . . . . . . . . . . . 32
2.5.2 Co-Clustering/Bi-Clustering . . . . . . . . . . . . . . . 33
2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3 Randomized Feature Selection 41
David J. Stracuzzi
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 Types of Randomizations . . . . . . . . . . . . . . . . . . . . 42
3.3 Randomized Complexity Classes . . . . . . . . . . . . . . . . 43
© 2008 by Taylor & Francis Group, LLC
3.4 Applying Randomization to Feature Selection . . . . . . . . 45
3.5 The Role of Heuristics . . . . . . . . . . . . . . . . . . . . . . 46
3.6 Examples of Randomized Selection Algorithms . . . . . . . . 47
3.6.1 A Simple Las Vegas Approach . . . . . . . . . . . . . 47
3.6.2 Two Simple Monte Carlo Approaches . . . . . . . . . 49
3.6.3 Random Mutation Hill Climbing . . . . . . . . . . . . 51
3.6.4 Simulated Annealing . . . . . . . . . . . . . . . . . . . 52
3.6.5 Genetic Algorithms . . . . . . . . . . . . . . . . . . . . 54
3.6.6 Randomized Variable Elimination . . . . . . . . . . . 56
3.7 Issues in Randomization . . . . . . . . . . . . . . . . . . . . 58
3.7.1 Pseudorandom Number Generators . . . . . . . . . . . 58
3.7.2 Sampling from Specialized Data Structures . . . . . . 59
3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4 Causal Feature Selection 63
Isabelle Guyon, Constantin Aliferis, and Andr´e Elisseeff
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.2 Classical “Non-Causal” Feature Selection . . . . . . . . . . . 65
4.3 The Concept of Causality . . . . . . . . . . . . . . . . . . . . 68
4.3.1 Probabilistic Causality . . . . . . . . . . . . . . . . . . 69
4.3.2 Causal Bayesian Networks . . . . . . . . . . . . . . . . 70
4.4 Feature Relevance in BayesianNetworks . . . . . . . . . . . 71
4.4.1 Markov Blanket . . . . . . . . . . . . . . . . . . . . . 72
4.4.2 Characterizing Features Selected via Classical Methods 73
4.5 CausalDiscovery Algorithms . . . . . . . . . . . . . . . . . . 77
4.5.1 A Prototypical Causal Discovery Algorithm . . . . . . 78
4.5.2 Markov Blanket Induction Algorithms . . . . . . . . . 79
4.6 Examples of Applications . . . . . . . . . . . . . . . . . . . . 80
4.7 Summary, Conclusions, and Open Problems . . . . . . . . . 82
II Extending Feature Selection 87
5 Active Learning of Feature Relevance 89
Emanuele Olivetti, Sriharsha Veeramachaneni, and Paolo Avesani
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2 Active Sampling for Feature Relevance Estimation . . . . . . 92
5.3 Derivation of the Sampling Benefit Function . . . . . . . . . 93
5.4 Implementation of the Active Sampling Algorithm . . . . . . 95
5.4.1 Data Generation Model: Class-Conditional Mixture of
Product Distributions . . . . . . . . . . . . . . . . . . 95
5.4.2 Calculation of Feature Relevances . . . . . . . . . . . 96
5.4.3 Calculation of Conditional Probabilities . . . . . . . . 97
5.4.4 Parameter Estimation . . . . . . . . . . . . . . . . . . 97
5.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.5.1 Synthetic Data . . . . . . . . . . . . . . . . . . . . . . 99
© 2008 by Taylor & Francis Group, LLC
5.5.2 UCI Datasets . . . . . . . . . . . . . . . . . . . . . . . 100
5.5.3 Computational Complexity Issues . . . . . . . . . . . 102
5.6 Conclusions and Future Work . . . . . . . . . . . . . . . . . 102
6 A Study of Feature Extraction Techniques Based on Decision
Border Estimate 109
Claudia Diamantini and Domenico Potena
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.1.1 Background on Statistical Pattern Classification . . . 111
6.2 Feature Extraction Based on Decision Boundary . . . . . . . 112
6.2.1 MLP-Based Decision Boundary Feature Extraction . . 113
6.2.2 SVM Decision Boundary Analysis . . . . . . . . . . . 114
6.3 Generalities About Labeled Vector Quantizers . . . . . . . . 115
6.4 Feature Extraction Based on Vector Quantizers . . . . . . . 116
6.4.1 Weighting of Normal Vectors . . . . . . . . . . . . . . 119
6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6.5.1 Experiment with Synthetic Data . . . . . . . . . . . . 122
6.5.2 Experiment with Real Data . . . . . . . . . . . . . . . 124
6.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7 Ensemble-Based Variable Selection Using Independent Probes
131
Eugene Tuv, Alexander Borisov, and Kari Torkkola
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.2 Tree Ensemble Methods in Feature Ranking . . . . . . . . . 132
7.3 The Algorithm: Ensemble-Based Ranking Against Independent
Probes . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.4.1 BenchmarkMethods . . . . . . . . . . . . . . . . . . . 138
7.4.2 Data and Experiments . . . . . . . . . . . . . . . . . . 139
7.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
8 Efficient Incremental-Ranked Feature Selection in Massive
Data 147
Roberto Ruiz, Jes´us S. Aguilar-Ruiz, and Jos´e C. Riquelme
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
8.2 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 148
8.3 Preliminary Concepts . . . . . . . . . . . . . . . . . . . . . . 150
8.3.1 Relevance . . . . . . . . . . . . . . . . . . . . . . . . . 150
8.3.2 Redundancy . . . . . . . . . . . . . . . . . . . . . . . . 151
8.4 Incremental Performance over Ranking . . . . . . . . . . . . 152
8.4.1 Incremental Ranked Usefulness . . . . . . . . . . . . . 153
8.4.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . 156
8.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
© 2008 by Taylor & Francis Group, LLC
III Weighting and Local Methods 167
9 Non-Myopic Feature Quality Evaluation with (R)ReliefF 169
Igor Kononenko and Marko Robnik ˇSikonja
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
9.2 FromImpurity to Relief . . . . . . . . . . . . . . . . . . . . . 170
9.2.1 ImpurityMeasures in Classification. . . . . . . . . . . 171
9.2.2 Relief for Classification . . . . . . . . . . . . . . . . . 172
9.3 ReliefF for Classification and RReliefF for Regression . . . . 175
9.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
9.4.1 ReliefF for Inductive Logic Programming . . . . . . . 178
9.4.2 Cost-Sensitive ReliefF . . . . . . . . . . . . . . . . . . 180
9.4.3 Evaluation of Ordered Features at Value Level . . . . 181
9.5 Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9.5.1 Difference of Probabilities . . . . . . . . . . . . . . . . 182
9.5.2 Portion of the Explained Concept . . . . . . . . . . . 183
9.6 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . 184
9.6.1 Time Complexity . . . . . . . . . . . . . . . . . . . . . 184
9.6.2 Active Sampling . . . . . . . . . . . . . . . . . . . . . 184
9.6.3 Parallelization . . . . . . . . . . . . . . . . . . . . . . 185
9.7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
9.7.1 Feature Subset Selection . . . . . . . . . . . . . . . . . 185
9.7.2 Feature Ranking . . . . . . . . . . . . . . . . . . . . . 186
9.7.3 FeatureWeighing . . . . . . . . . . . . . . . . . . . . . 186
9.7.4 Building Tree-BasedModels . . . . . . . . . . . . . . . 187
9.7.5 Feature Discretization . . . . . . . . . . . . . . . . . . 187
9.7.6 Association Rules and Genetic Algorithms . . . . . . . 187
9.7.7 Constructive Induction . . . . . . . . . . . . . . . . . . 188
9.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
10 Weighting Method for Feature Selection in K-Means 193
Joshua Zhexue Huang, Jun Xu, Michael Ng, and Yunming Ye
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
10.2 Feature Weighting in k-Means . . . . . . . . . . . . . . . . . 194
10.3 W-k-Means Clustering Algorithm . . . . . . . . . . . . . . . 197
10.4 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . 198
10.5 Subspace Clustering with k-Means . . . . . . . . . . . . . . . 200
10.6 Text Clustering . . . . . . . . . . . . . . . . . . . . . . . . . 201
10.6.1 Text Data and Subspace Clustering . . . . . . . . . . 202
10.6.2 Selection of KeyWords . . . . . . . . . . . . . . . . . 203
10.7 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 204
10.8 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
© 2008 by Taylor & Francis Group, LLC
11 Local Feature Selection for Classification 211
Carlotta Domeniconi and Dimitrios Gunopulos
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
11.2 The Curse of Dimensionality . . . . . . . . . . . . . . . . . . 213
11.3 AdaptiveMetric Techniques . . . . . . . . . . . . . . . . . . 214
11.3.1 Flexible Metric Nearest Neighbor Classification . . . . 215
11.3.2 Discriminant Adaptive Nearest Neighbor Classification 216
11.3.3 Adaptive Metric Nearest Neighbor Algorithm . . . . . 217
11.4 Large Margin Nearest Neighbor Classifiers . . . . . . . . . . 222
11.4.1 Support Vector Machines . . . . . . . . . . . . . . . . 223
11.4.2 FeatureWeighting . . . . . . . . . . . . . . . . . . . . 224
11.4.3 Large Margin Nearest Neighbor Classification . . . . . 225
11.4.4 Weighting Features Increases the Margin . . . . . . . 227
11.5 Experimental Comparisons . . . . . . . . . . . . . . . . . . . 228
11.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
12 Feature Weighting through Local Learning 233
Yijun Sun
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
12.2 Mathematical Interpretation of Relief . . . . . . . . . . . . . 235
12.3 Iterative Relief Algorithm . . . . . . . . . . . . . . . . . . . . 236
12.3.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 236
12.3.2 Convergence Analysis . . . . . . . . . . . . . . . . . . 238
12.4 Extension to Multiclass Problems . . . . . . . . . . . . . . . 240
12.5 Online Learning . . . . . . . . . . . . . . . . . . . . . . . . . 240
12.6 Computational Complexity . . . . . . . . . . . . . . . . . . . 242
12.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
12.7.1 Experimental Setup . . . . . . . . . . . . . . . . . . . 242
12.7.2 Experiments on UCI Datasets . . . . . . . . . . . . . . 244
12.7.3 Choice of KernelWidth . . . . . . . . . . . . . . . . . 248
12.7.4 Online Learning . . . . . . . . . . . . . . . . . . . . . 248
12.7.5 Experiments on Microarray Data . . . . . . . . . . . . 249
12.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
IV Text Classification and Clustering 255
13 Feature Selection for Text Classification 257
George Forman
13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
13.1.1 Feature Selection Phyla . . . . . . . . . . . . . . . . . 259
13.1.2 Characteristic Difficulties of Text Classification Tasks 260
13.2 Text Feature Generators . . . . . . . . . . . . . . . . . . . . 261
13.2.1 WordMerging . . . . . . . . . . . . . . . . . . . . . . 261
13.2.2 Word Phrases . . . . . . . . . . . . . . . . . . . . . . . 262
13.2.3 Character N-grams . . . . . . . . . . . . . . . . . . . . 263
© 2008 by Taylor & Francis Group, LLC
13.2.4 Multi-Field Records . . . . . . . . . . . . . . . . . . . 264
13.2.5 Other Properties . . . . . . . . . . . . . . . . . . . . . 264
13.2.6 Feature Values . . . . . . . . . . . . . . . . . . . . . . 265
13.3 Feature Filtering for Classification . . . . . . . . . . . . . . . 265
13.3.1 Binary Classification . . . . . . . . . . . . . . . . . . . 266
13.3.2 Multi-Class Classification . . . . . . . . . . . . . . . . 269
13.3.3 Hierarchical Classification . . . . . . . . . . . . . . . . 270
13.4 Practical and Scalable Computation . . . . . . . . . . . . . . 271
13.5 A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . 272
13.6 Conclusion and FutureWork . . . . . . . . . . . . . . . . . . 274
14 A Bayesian Feature Selection Score Based on Na¨ıve Bayes
Models 277
Susana Eyheramendy and David Madigan
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
14.2 Feature Selection Scores . . . . . . . . . . . . . . . . . . . . . 279
14.2.1 Posterior Inclusion Probability (PIP) . . . . . . . . . . 280
14.2.2 Posterior Inclusion Probability (PIP) under a Bernoulli
distribution . . . . . . . . . . . . . . . . . . . . . . . . 281
14.2.3 Posterior Inclusion Probability (PIPp) under Poisson
distributions . . . . . . . . . . . . . . . . . . . . . . . 283
14.2.4 Information Gain (IG) . . . . . . . . . . . . . . . . . . 284
14.2.5 Bi-Normal Separation (BNS) . . . . . . . . . . . . . . 285
14.2.6 Chi-Square . . . . . . . . . . . . . . . . . . . . . . . . 285
14.2.7 Odds Ratio . . . . . . . . . . . . . . . . . . . . . . . . 286
14.2.8 Word Frequency . . . . . . . . . . . . . . . . . . . . . 286
14.3 Classification Algorithms . . . . . . . . . . . . . . . . . . . . 286
14.4 Experimental Settings and Results . . . . . . . . . . . . . . . 287
14.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 287
14.4.2 Experimental Results . . . . . . . . . . . . . . . . . . 288
14.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
15 Pairwise Constraints-Guided Dimensionality Reduction 295
Wei Tang and Shi Zhong
15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
15.2 Pairwise Constraints-Guided Feature Projection . . . . . . . 297
15.2.1 Feature Projection . . . . . . . . . . . . . . . . . . . . 298
15.2.2 Projection-Based Semi-supervised Clustering . . . . . 300
15.3 Pairwise Constraints-Guided Co-clustering . . . . . . . . . . 301
15.4 Experimental Studies . . . . . . . . . . . . . . . . . . . . . . 302
15.4.1 Experimental Study – I . . . . . . . . . . . . . . . . . 302
15.4.2 Experimental Study – II . . . . . . . . . . . . . . . . . 306
15.4.3 Experimental Study – III . . . . . . . . . . . . . . . . 309
15.5 Conclusion and FutureWork . . . . . . . . . . . . . . . . . . 310
© 2008 by Taylor & Francis Group, LLC
16 Aggressive Feature Selection by Feature Ranking 313
Masoud Makrehchi and Mohamed S. Kamel
16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
16.2 Feature Selection by Feature Ranking . . . . . . . . . . . . . 314
16.2.1 Multivariate Characteristic of Text Classifiers . . . . . 316
16.2.2 Term Redundancy . . . . . . . . . . . . . . . . . . . . 316
16.3 Proposed Approach to Reducing Term Redundancy . . . . . 320
16.3.1 Stemming, Stopwords, and Low-DF Terms Elimination 320
16.3.2 Feature Ranking . . . . . . . . . . . . . . . . . . . . . 320
16.3.3 Redundancy Reduction . . . . . . . . . . . . . . . . . 322
16.3.4 Redundancy Removal Algorithm . . . . . . . . . . . . 325
16.3.5 Term Redundancy Tree . . . . . . . . . . . . . . . . . 326
16.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . 326
16.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
V Feature Selection in Bioinformatics 335
17 Feature Selection for Genomic Data Analysis 337
Lei Yu
17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
17.1.1 Microarray Data and Challenges . . . . . . . . . . . . 337
17.1.2 Feature Selection for Microarray Data . . . . . . . . . 338
17.2 Redundancy-Based Feature Selection . . . . . . . . . . . . . 340
17.2.1 Feature Relevance and Redundancy . . . . . . . . . . 340
17.2.2 An Efficient Framework for Redundancy Analysis . . . 343
17.2.3 RBF Algorithm . . . . . . . . . . . . . . . . . . . . . . 345
17.3 Empirical Study . . . . . . . . . . . . . . . . . . . . . . . . . 347
17.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 347
17.3.2 Experimental Settings . . . . . . . . . . . . . . . . . . 349
17.3.3 Results and Discussion . . . . . . . . . . . . . . . . . . 349
17.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
18 A Feature Generation Algorithm with Applications to Biological
Sequence Classification 355
Rezarta Islamaj Dogan, Lise Getoor, and W. John Wilbur
18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
18.2 Splice-Site Prediction . . . . . . . . . . . . . . . . . . . . . . 356
18.2.1 The Splice-Site Prediction Problem . . . . . . . . . . . 356
18.2.2 Current Approaches . . . . . . . . . . . . . . . . . . . 357
18.2.3 Our Approach . . . . . . . . . . . . . . . . . . . . . . 359
18.3 Feature Generation Algorithm . . . . . . . . . . . . . . . . . 359
18.3.1 Feature Type Analysis . . . . . . . . . . . . . . . . . . 360
18.3.2 Feature Selection . . . . . . . . . . . . . . . . . . . . 362
18.3.3 Feature Generation Algorithm (FGA) . . . . . . . . . 364
18.4 Experiments and Discussion . . . . . . . . . . . . . . . . . . 366
© 2008 by Taylor & Francis Group, LLC
18.4.1 Data Description . . . . . . . . . . . . . . . . . . . . 366
18.4.2 Feature Generation . . . . . . . . . . . . . . . . . . . . 367
18.4.3 Prediction Results for Individual Feature Types . . . . 369
18.4.4 Splice-Site Prediction with FGA Features . . . . . . . 370
18.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
19 An Ensemble Method for Identifying Robust Features for
Biomarker Discovery 377
Diana Chan, Susan M. Bridges, and Shane C. Burgess
19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
19.2 Biomarker Discovery from Proteome Profiles . . . . . . . . . 378
19.3 Challenges of Biomarker Identification . . . . . . . . . . . . . 380
19.4 Ensemble Method for Feature Selection . . . . . . . . . . . . 381
19.5 Feature Selection Ensemble . . . . . . . . . . . . . . . . . . . 383
19.6 Results and Discussion . . . . . . . . . . . . . . . . . . . . . 384
19.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
20 Model Building and Feature Selection with Genomic Data 393
Hui Zou and Trevor Hastie
20.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
20.2 Ridge Regression, Lasso, and Bridge . . . . . . . . . . . . . . 394
20.3 Drawbacks of the Lasso . . . . . . . . . . . . . . . . . . . . . 396
20.4 The Elastic Net . . . . . . . . . . . . . . . . . . . . . . . . . 397
20.4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . 397
20.4.2 A Stylized Example . . . . . . . . . . . . . . . . . . . 399
20.4.3 Computation and Tuning . . . . . . . . . . . . . . . . 400
20.4.4 Analyzing the Cardiomypathy Data . . . . . . . . . . 402
20.5 The Elastic-Net Penalized SVM . . . . . . . . . . . . . . . . 404
20.5.1 Support Vector Machines . . . . . . . . . . . . . . . . 404
20.5.2 A New SVM Classifier . . . . . . . . . . . . . . . . . . 405
20.6 Sparse Eigen-Genes . . . . . . . . . . . . . . . . . . . . . . . 407
20.6.1 PCA and Eigen-Genes . . . . . . . . . . . . . . . . . . 408
20.6.2 Sparse Principal Component Analysis . . . . . . . . . 408
20.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409

:11bb
[hide]
[/hide]

xuexi...................................................

新年的好书，新书也是。。。:16bb :16bb

:30bb :30bb :30bb :30bb

这么快啊

:27bb :27bb :27bb :27bb

学习一下，增长见识，感谢分享！！

学习一下，增长见识，感谢分享！！

新书都顶！:))1 :))1 :))1

:27bb :27bb :27bb :30bb :30bb :30bb

:11bb :11bb :11bb

08新书阿
:27bb :27bb

thinks，thinks，thinks

新书，不错！:30bb :30bb :30bb

好书，新书，谢谢！:31bb

下来看看，学习下

:27bb :27bb :27bb

:11bb :27bb :30bb

新書
謝謝芬想囉 :11bb :27bb

感谢楼主分享
:11bb :11bb :11bb :11bb

:27bb 4# 00d44

谢谢楼主的分享，

:27bb：17de：13de

下下来瞧瞧

没看到下载的地方呢，是不是要回复才能看到

Thank you for your sharing

看看。。。。。。。

Thank you for your sharing

gooooooooood

学习一下，增长见识
下来看看

好书！下来看看，学习下

感谢楼主，感谢楼主

象是一本专著，可以看看!

     下来看看，学习下

我回复乐乐乐乐乐乐

啥也不说了，楼主就是给力！