Neural network methods for natural language processing

Neural networks are a family of powerful machine learning models. This book focuses on the application of neural network models to natural language data. The first half of the book (Parts I and II) covers the basics of supervised machine learning and feed-forward neural networks, the basics of working with machine learning over language data, and the use of vector-based rather than symbolic representations for words. It also covers the computation-graph abstraction, which allows to easily define and train arbitrary neural networks, and is the basis behind the design of contemporary neural network software libraries. The second part of the book (Parts III and IV) introduces more specialized neural network architectures, including 1D convolutional neural networks, recurrent neural networks, conditioned-generation models, and attention-based models. These architectures and techniques are the driving force behind state-of-the-art algorithms for machine translation, syntactic parsing, and many other applications. Finally, we also discuss tree-shaped networks, structured prediction, and the prospects of multi-task learning.;1. Introduction -- 2. Learning basics and linear models -- 3. From linear models to multi-layer perceptrons -- 4. Feed-forward neural networks -- 5. Neural network training -- 6. Features for textual data -- 7. Case studies of NLP features -- 8. From textual features to inputs -- 9. Language modeling -- 10. Pre-trained word representations -- 11. Using word embeddings -- 12. Case study: a feed-forward architecture for sentence meaning inference -- 13. Ngram detectors: convolutional neural networks -- 14. Recurrent neural networks: modeling sequences and stacks -- 15. Concrete recurrent neural network architectures -- 16. Modeling with recurrent networks -- 17. Conditioned generation -- 18. Modeling trees with recursive neural networks -- 19. Structured output prediction -- 20. Cascaded, multi-task and semi-supervised learning -- 21. Conclusion -- Bibliography -- Author's biography. Human Language Technologies Preface 19 Acknowledgments 23 Introduction 25 The Challenges of Natural Language Processing 25 Neural Networks and Deep Learning 26 Deep Learning in NLP 27 Success Stories 28 Coverage and Organization 30 What's not Covered 32 A Note on Terminology 32 Mathematical Notation 33 Supervised Classification and Feed-forward Neural Networks 35 Learning Basics and Linear Models 37 Learning Basics and Linear Models 37 Supervised Learning and Parameterized Functions 37 Train, Test, and Validation Sets 38 Linear Models 40 Binary Classification 40 Log-linear Binary Classification 44 Multi-class Classification 44 Representations 45 One-Hot and Dense Vector Representations 46 Log-linear Multi-class Classification 48 Training as Optimization 49 Loss Functions 50 Regularization 52 Gradient-based Optimization 54 Stochastic Gradient Descent 55 Worked-out Example 57 Beyond SGD 59 From Linear Models to Multi-layer Perceptrons 61 Limitations of Linear Models: The XOR Problem 61 Nonlinear Input Transformations 62 Kernel Methods 62 Trainable Mapping Functions 63 Feed-forward Neural Networks 65 A Brain-inspired Metaphor 65 In Mathematical Notation 67 Representation Power 68 Common Nonlinearities 69 Loss Functions 70 Regularization and Dropout 71 Similarity and Distance Layers 72 Embedding Layers 73 Neural Network Training 75 The Computation Graph Abstraction 75 Forward Computation 77 Backward Computation (Derivatives, Backprop) 77 Software 79 Implementation Recipe 81 Network Composition 82 Practicalities 82 Choice of Optimization Algorithm 82 Initialization 83 Restarts and Ensembles 83 Vanishing and Exploding Gradients 84 Saturation and Dead Neurons 84 Shuffling 85 Learning Rate 85 Minibatches 85 Working with Natural Language Data 87 Features for Textual Data 89 Typology of NLP Classification Problems 89 Features for NLP Problems 91 Directly Observable Properties 91 Inferred Linguistic Properties 94 Core Features vs. Combination Features 98 Ngram Features 99 Distributional Features 100 Case Studies of NLP Features 101 Document Classification: Language Identification 101 Document Classification: Topic Classification 101 Document Classification: Authorship Attribution 102 Word-in-context: Part of Speech Tagging 103 Word-in-context: Named Entity Recognition 105 Word in Context, Linguistic Features: Preposition Sense Disambiguation 106 Relation Between Words in Context: Arc-Factored Parsing 109 From Textual Features to Inputs 113 Encoding Categorical Features 113 One-hot Encodings 113 Dense Encodings (Feature Embeddings) 114 Dense Vectors vs. One-hot Representations 114 Combining Dense Vectors 116 Window-based Features 117 Variable Number of Features: Continuous Bag of Words 117 Relation Between One-hot and Dense Vectors 118 Odds and Ends 119 Distance and Position Features 119 Padding, Unknown Words, and Word Dropout 120 Feature Combinations 121 Vector Sharing 122 Dimensionality 123 Embeddings Vocabulary 123 Network's Output 123 Example: Part-of-Speech Tagging 124 Example: Arc-factored Parsing 125 Language Modeling 129 The Language Modeling Task 129 Evaluating Language Models: Perplexity 130 Traditional Approaches to Language Modeling 131 Further Reading 132 Limitations of Traditional Language Models 132 Neural Language Models 133 Using Language Models for Generation 136 Byproduct: Word Representations 137 Pre-trained Word Representations 139 Pre-trained Word Representations 139 Random Initialization 139 Supervised Task-specific Pre-training 139 Unsupervised Pre-training 140 Using Pre-trained Embeddings 141 Word Embedding Algorithms 141 Distributional Hypothesis and Word Representations 142 From Neural Language Models to Distributed Representations 146 Connecting the Worlds 149 Other Algorithms 150 The Choice of Contexts 151 Window Approach 151 Sentences, Paragraphs, or Documents 152 Syntactic Window 153 Multilingual 154 Character-based and Sub-word Representations 155 Dealing with Multi-word Units and Word Inflections 156 Limitations of Distributional Methods 157 Using Word Embeddings 159 Obtaining Word Vectors 159 Word Similarity 159 Word Clustering 160 Finding Similar Words 160 Similarity to a Group of Words 160 Odd-one Out 161 Short Document Similarity 161 Word Analogies 162 Retrofitting and Projections 163 Practicalities and Pitfalls 164 Case Study: A Feed-forward Architecture for Sentence 165 Case Study: A Feed-forward Architecture for Sentence Meaning Inference 165 Natural Language Inference and the SNLI Dataset 165 A Textual Similarity Network 166 Specialized Architectures 171 Ngram Detectors: Convolutional Neural Networks 175 Basic Convolution + Pooling 176 1D Convolutions Over Text 177 Vector Pooling 179 Variations 182 Alternative: Feature Hashing 182 Hierarchical Convolutions 183 Recurrent Neural Networks: Modeling Sequences and Stacks 187 The RNN Abstraction 187 RNN Training 190 Common RNN Usage-patterns 191 Acceptor 191 Encoder 191 Transducer 192 Bidirectional RNNs (biRNN) 193 Multi-layer (stacked) RNNs 195 RNNs for Representing Stacks 196 A Note on Reading the Literature 197 Concrete Recurrent Neural Network Architectures 201 CBOW as an RNN 201 Simple RNN 201 Gated Architectures 202 LSTM 203 GRU 205 Other Variants 206 Dropout in RNNs 207 Modeling with Recurrent Networks 209 Modeling with Recurrent Networks 209 Acceptors 209 Sentiment Classification 209 Subject-verb Agreement Grammaticality Detection 211 RNNs as Feature Extractors 213 Part-of-speech Tagging 213 RNN–CNN Document Classification 215 Arc-factored Dependency Parsing 216 Conditioned Generation 219 RNN Generators 219 Training Generators 220 Conditioned Generation (Encoder-Decoder) 220 Sequence to Sequence Models 222 Applications 224 Other Conditioning Contexts 226 Unsupervised Sentence Similarity 227 Conditioned Generation with Attention 228 Computational Complexity 232 Interpretability 232 Attention-based Models in NLP 232 Machine Translation 232 Morphological Inflection 234 Syntactic Parsing 235 Additional Topics 237 Modeling Trees with Recursive Neural Networks 239 Modeling Trees with Recursive Neural Networks 239 Formal Definition 239 Extensions and Variations 242 Training Recursive Neural Networks 243 A Simple Alternative–Linearized Trees 243 Outlook 243 Structured Output Prediction 245 Search-based Structured Prediction 245 Structured Prediction with Linear Models 245 Nonlinear Structured Prediction 246 Probabilistic Objective (CRF) 248 Approximate Search 248 Reranking 249 See Also 249 Greedy Structured Prediction 250 Conditional Generation as Structured Output Prediction 251 Examples 251 Search-based Structured Prediction: First-order Dependency Parsing 252 Neural-CRF for Named Entity Recognition 253 Approximate NER-CRF With Beam-Search 256 Cascaded, Multi-task and Semi-supervised Learning 259 Model Cascading 259 Multi-task Learning 262 Training in a Multi-task Setup 265 Selective Sharing 266 Word-embeddings Pre-training as Multi-task Learning 267 Multi-task Learning in Conditioned Generation 267 Multi-task Learning as Regularization 267 Caveats 267 Semi-supervised Learning 268 Examples 269 Gaze-prediction and Sentence Compression 269 Arc Labeling and Syntactic Parsing 270 Preposition Sense Disambiguation and Preposition Translation Prediction 271 Conditioned Generation: Multilingual Machine Translation, Parsing, and Image Captioning 272 Outlook 273 Conclusion 275 What Have We Seen? 275 The Challenges Ahead 275 Bibliography 277 Author's Biography 311 Blank Page 2 natural language processing,machine learning,supervised learning,deep learning,neural networks,word embeddings,recurrent neural networks,sequence to sequence models

Neural networks are a family of powerful machine learning models and this book focuses on their application to natural language data.

The first half of the book (Parts I and II) covers the basics of supervised machine learning and feed-forward neural networks, the basics of working with machine learning over language data, and the use of vector-based rather than symbolic representations for words. It also covers the computation-graph abstraction, which allows to easily define and train arbitrary neural networks, and is the basis behind the design of contemporary neural network software libraries.

The second part of the book (Parts III and IV) introduces more specialized neural network architectures, including 1D convolutional neural networks, recurrent neural networks, conditioned-generation models, and attention-based models. These architectures and techniques are the driving force behind state-of-the-art algorithms for machine translation, syntactic parsing, and many other applications. Finally, we also discuss tree-shaped networks, structured prediction, and the prospects of multi-task learning.