Responsible Graph Neural Networks

More frequent and complex cyber threats require robust, automated, and rapid responses from cyber-security specialists. This book offers a complete study in the area of graph learning in cyber, emphasizing graph neural networks (GNNs) and their cyber-security applications. Three parts examine the basics, methods and practices, and advanced topics. The first part presents a grounding in graph data structures and graph embedding and gives a taxonomic view of GNNs and cyber-security applications. The second part explains three different categories of graph learning, including deterministic, generative, and reinforcement learning and how they can be used for developing cyber defense models. The discussion of each category covers the applicability of simple and complex graphs, scalability, representative algorithms, and technical details. Undergraduate students, graduate students, researchers, cyber analysts, and AI engineers looking to understand practical deep learning methods will find this book an invaluable resource. This book offers a complete study in the area of graph learning in cyber, emphasising graph neural networks (GNNs) and their cyber security applications. Cover 1 Half Title 2 Title Page 4 Copyright Page 5 Dedication 6 Contents 8 Preface 14 How to Use This Book 16 1. Introduction to Graph Intelligence 18 1.1. Introduction 18 1.2. Feedforward Neural Network (FFNN) 19 1.2.1. Architecture 19 1.2.2. Activation Functions 21 1.2.2.1. Binary Step Function 21 1.2.2.2. Linear Activation Function 22 1.2.2.3. Nonlinear Activation Functions 22 1.3. Convolutional Neural Networks (CNNs) 25 1.3.1. Convolutional Layer 25 1.3.2. Pooling Layers 26 1.3.2.1. Max Pooling 27 1.3.2.2. Average Pooling 27 1.4. Recurrent Neural Networks (RNNs) 29 1.4.1. Vanilla RNN 30 1.4.2. Long Short-Term Memory (LSTM) 31 1.4.3. Gated Recurrent Units (GRUs) 33 1.4.4. Bidirectional Recurrent Neural Network (Bi-RNN) 35 1.5. Autoencoder 37 1.6. Deep Learning for Graph Intelligence 39 1.7. What Is Covered in This Book? 40 1.8. Case Study 42 References 43 2. Fundamentals of Graph Representations 44 2.1. Introduction 44 2.2. Graph Representation 45 2.3. Properties and Measure 46 2.3.1. Degree 47 2.3.2. Connectivity 48 2.3.3. Centrality 50 2.3.3.1. Degree Centrality 50 2.3.3.2. Eigenvector Centrality 51 2.3.3.3. Katz Centrality 52 2.3.3.4. Betweenness Centrality 53 2.3.3.5. Closeness Centrality 56 2.3.3.6. Harmonic Centrality 57 2.4. Spectral Graph Analysis 58 2.4.1. Laplacian Matrix 58 2.4.2. Graph Laplacian Matrix: On Eigenvalues 60 2.5. Graph Signal Analysis 61 2.5.1. Graph Fourier Transform 61 2.6. Complex Graphs 63 2.6.1. Bipartite Graphs 63 2.6.2. Heterogeneous Graphs 64 2.6.3. Multi-dimensional Graphs 67 2.6.4. Signed Graphs 68 2.6.5. Hypergraphs 68 2.6.6. Dynamic Graphs 69 2.7. Graph Intelligence Tasks 69 2.7.1. Graph-Oriented Tasks 70 2.7.1.1. Graph Classification 70 2.7.2. Node-Oriented Tasks 70 2.7.2.1. Node Classification 71 2.7.2.2. Link Prediction 71 2.8. Case Study 72 References 72 3. Graph Embedding: Methods, Taxonomies, and Applications 74 3.1. Introduction 74 3.2. Homogeneous Graph Embedding 76 3.2.1. Node Co-occurrence 76 3.2.2. Node State 87 3.2.3. Community Topology 88 3.3. Heterogeneous Graph Embedding 90 3.3.1. Application-Based Heterogeneous Graph Embedding 92 3.3.2. Feature-Based Heterogeneous Graph Embedding 93 3.3.3. Topology-Retained Heterogeneous Graph Embedding 94 3.3.3.1. Edge-Based Embedding 94 3.3.3.2. Path-Based Embedding 96 3.3.3.3. Subgraph Embedding 99 3.3.4. Dynamic Heterogeneous Graph Embedding 100 3.4. Bipartite Graph Embedding 101 3.5. Case Study 101 References 102 4. Toward Graph Neural Networks: Essentials and Pillars 106 4.1. Introduction 106 4.2. Graph Filters 107 4.2.1. Spatial-Dependent Graph Filters 107 4.2.2. Spectral-Dependent Graph Filters 107 4.3. Graph Normalization 108 4.3.1. Batch Normalization 109 4.3.2. Instance Normalization 109 4.3.3. Layer Normalization 109 4.3.4. Graph Normalization 109 4.3.5. Graph Size Normalization 110 4.3.6. Pair Normalization 110 4.3.7. Mean Subtraction Normalization 110 4.3.8. Message Normalization 110 4.3.9. Differentiable Group Normalization 110 4.4. Graph Pooling 111 4.4.1. Global Add Pooling 111 4.4.2. Global Mean Pooling 112 4.4.3. Global Max Pooling 112 4.4.4. topk Pooling 113 4.4.5. Self-Attention (SA) Graph Pooling 114 4.4.6. Sort Pooling 114 4.4.7. Edge Pooling 115 4.4.8. Adaptive Structure Aware Pooling (ASAP) 115 4.4.9. PAN Pooling 116 4.4.10. Memory Pooling 116 4.4.11. Differentiable Pooling 116 4.4.12. MinCut Pooling 117 4.4.13. Spectral Modularity Pooling 117 4.5. Graph Aggregation 118 4.5.1. Sum Aggregation 119 4.5.2. Mean Aggregation 119 4.5.3. Max Aggregation 120 4.5.4. Min Aggregation 120 4.5.5. Multiple Aggregation 120 4.5.6. Variance Aggregation 120 4.5.7. Standard Deviation (STD) Aggregation 121 4.5.8. SoftMax Aggregation 121 4.5.9. Power Mean Aggregation 121 4.5.10. Long Short-term Memory (LSTM) Aggregation 121 4.5.11. Set2Set 122 4.5.12. Degree Scaler Aggregation 123 4.5.13. Graph Multiset Transformer 124 4.5.14. Attentional Aggregation 124 4.6. Case Study 124 References 125 5. Graph Convolution Networks: A Journey from Start to End 128 5.1. Introduction 128 5.2. Graph Convolutional Network 129 5.3. Deeper Graph Convolution Network 133 5.4. GCN with Initial Residual and Identity Mapping (GCNII) 136 5.5. Topology Adaptive Graph Convolutional Networks 139 5.6. Relational Graph Convolutional Network 142 5.7. Case Study 145 References 146 6. Graph Attention Networks: A Journey from Start to End 148 6.1. Introduction 148 6.2. Graph Attention Network 148 6.3. Graph Attention Network version 2(GATv2) 154 6.4. Generalized Graph Transformer Network 159 6.5. Graph Transformer Network (GTN) 168 6.6. Case Study 173 References 173 7. Recurrent Graph Neural Networks: A Journey from Start to End 176 7.1. Introduction 176 7.2. Tree-Long Short-Term Memory 177 7.2.1. Child-Sum Tree-LSTMs 177 7.2.2. N-ary Tree-LSTMs 179 7.3. Gated Graph Sequence Neural Networks 181 7.3.1. Graph Classification 184 7.3.2. Node Section 186 7.3.3. Sequence Outputs 187 7.4. Graph-Gated Recurrent Units 189 7.5. Case Study 192 References 193 8. Graph Autoencoders: A Journey from Start to End 194 8.1. Introduction 194 8.2. General Framework of Graph Autoencoders 195 8.3. Variational Graph Autoencoder 200 8.4. Regularized Variational Graph Autoencoder 204 8.5. Graphite Variational Autoencoder 206 8.6. Dirichlet Graph Variational Autoencoder (DGVAE) 207 8.7. Case Study 210 References 210 9. Interpretable Graph Intelligence: A Journey from Black to White Box 212 9.1. Introduction 212 9.2. Interpretability Methods for Graph Intelligence 214 9.3. Instance-Level Interpretability 215 9.3.1. Gradients-Dependent Explanations 216 9.3.1.1. Conceptual View 216 9.3.1.2. Methods 216 9.3.2. Perturbation-Dependent Explanation Methods 220 9.3.2.1. Conceptual View 220 9.3.2.2.. Explanation Methods 222 9.3.3. Surrogate Models 247 9.3.3.1. A Conceptual View 247 9.3.3.2. Surrogate Interpretability Methods 248 9.3.4. Decomposition Explanation 265 9.3.4.1. Conceptual View 265 9.3.4.2. Decomposition Methods 267 9.4. Model-Level Explanations 268 9.5. Interpretability Evaluation Metrics 269 9.5.1. Fidelity Measure 269 9.5.2. Sparsity Measure 271 9.5.3. Stability Measure 274 9.6. Case Study 274 References 274 10. Toward Privacy Preserved Graph Intelligence: Concepts, Methods, and Applications 278 10.1. Introduction 278 10.2. Privacy Threats for Graph Intelligence 280 10.3. Threat Models of Privacy Attacks 283 10.3.1. Methods of Privacy Attack on GNNs 284 10.4. Differential Privacy for Graph Intelligence 286 10.5. Federated Graph Intelligence 291 10.5.1. Horizontal FL 297 10.5.2. Vertical FL 300 10.5.3. Federated Transfer Learning (FTL) 303 10.6. Open-Source Frameworks 304 10.6.1. TensorFlow Federated 305 10.6.2. FedML 305 10.6.3. Federated AI Technology Enabler (FATE) 306 10.6.4. IBM Federated Learning 306 10.6.5. Flower 306 10.6.6. Leaf 307 10.6.7. NVIDIA Federated Learning Application Runtime Environment (NVIDIA FLARE) 307 10.6.8. OpenFL 308 10.6.9. PaddleFL 308 10.6.10. PySyft and PyGrid 308 10.6.11. Sherpa.ai 309 10.7. Case Study 310 References 310 Index 314 Cyber;,Adversarial,Methods;,Graph;,Security;,Graph,Theory;,Neural,Networks Cyber,Adversarial Methods,Graph,Security,Graph Theory,Neural Networks