Lagged Exact Bayesian Online Changepoint Detection

Identifying changes in the generative process of sequential data, known as changepoint detection, has become an increasingly important topic for a wide variety of fields. A recently developed approach, which we call EXact Online Bayesian Changepoint Detection (EXO), has shown reasonable results with efficient computation for real time updates. However, when the changes are relatively small, EXO starts to have difficulty in detecting changepoints accurately. We propose a new algorithm called \ell-Lag EXact Online Bayesian Changepoint Detection (LEXO-\ell), which improves the accuracy of the detection by incorporating \ell time lags in the inference. We prove that LEXO-1 finds the exact posterior distribution for the current run length and can be computed efficiently, with extension to arbitrary lag. Additionally, we show that LEXO-1 performs better than EXO in an extensive simulation study; this study is extended to higher order lags to illustrate the performance of the generalized methodology. Lastly, we illustrate applicability with two real world data examples comparing EXO and LEXO-1.

Checkpoint Ensembles: Ensemble Methods from a Single Training Process

We present the checkpoint ensembles method that can learn ensemble models on a single training process. Although checkpoint ensembles can be applied to any parametric iterative learning technique, here we focus on neural networks. Neural networks’ composable and simple neurons make it possible to capture many individual and interaction effects among features. However, small sample sizes and sampling noise may result in patterns in the training data that are not representative of the true relationship between the features and the outcome. As a solution, regularization during training is often used (e.g. dropout). However, regularization is no panacea — it does not perfectly address overfitting. Even with methods like dropout, two methodologies are commonly used in practice. First is to utilize a validation set independent to the training set as a way to decide when to stop training. Second is to use ensemble methods to further reduce overfitting and take advantage of local optima (i.e. averaging over the predictions of several models). In this paper, we explore checkpoint ensembles — a simple technique that combines these two ideas in one training process. Checkpoint ensembles improve performance by averaging the predictions from ‘checkpoints’ of the best models within single training process. We use three real-world data sets — text, image, and electronic health record data — using three prediction models: a vanilla neural network, a convolutional neural network, and a long short term memory network to show that checkpoint ensembles outperform existing methods: a method that selects a model by minimum validation score, and two methods that average models by weights. Our results also show that checkpoint ensembles capture a portion of the performance gains that traditional ensembles provide.

An Extension of Deep Pathway Analysis: A Pathway Route Analysis Framework Incorporating Multi-dimensional Cancer Genomics Data

Recent breakthroughs in cancer research have come via the up-and-coming field of pathway analysis. By applying statistical methods to prior known gene and protein regulatory information, pathway analysis provides a meaningful way to interpret genomic data. While many gene/protein regulatory relationships have been studied, never before has such a significant amount data been made available in organized forms of gene/protein regulatory networks and pathways. However, pathway analysis research is still in its infancy, especially when applying it to solve practical problems. In this paper we propose a new method of studying biological pathways, one that cross analyzes mutation information, transcriptome and proteomics data. Using this outcome, we identify routes of aberrant pathways potentially responsible for the etiology of disease. Each pathway route is encoded as a bayesian network which is initialized with a sequence of conditional probabilities specifically designed to encode directionality of regulatory relationships encoded in the pathways. Far more complex interactions, such as phosphorylation and methylation, among others, in the pathways can be modeled using this approach. The effectiveness of our model is demonstrated through its ability to distinguish real pathways from decoys on TCGA mRNA-seq, mutation, Copy Number Variation and phosphorylation data for both Breast cancer and Ovarian cancer study. The majority of pathways distinguished can be confirmed by biological literature. Moreover, the proportion of correctly indentified pathways is \% higher than previous work where only mRNA-seq mutation data is incorporated for breast cancer patients. Consequently, such an in-depth pathway analysis incorporating more diverse data can give rise to the accuracy of perturbed pathway detection.

iVQA: Inverse Visual Question Answering

In recent years, visual question answering (VQA) has become topical as a long-term goal to drive computer vision and multi-disciplinary AI research. The premise of VQA’s significance, is that both the image and textual question need to be well understood and mutually grounded in order to infer the correct answer. However, current VQA models perhaps `understand’ less than initially hoped, and instead master the easier task of exploiting cues given away in the question and biases in the answer distribution. In this paper we propose the inverse problem of VQA (iVQA), and explore its suitability as a benchmark for visuo-linguistic understanding. The iVQA task is to generate a question that corresponds to a given image and answer pair. Since the answers are less informative than the questions, and the questions have less learnable bias, an iVQA model needs to better understand the image to be successful. We pose question generation as a multi-modal dynamic inference process and propose an iVQA model that can gradually adjust its focus of attention guided by both a partially generated question and the answer. For evaluation, apart from existing linguistic metrics, we propose a new ranking metric. This metric compares the ground truth question’s rank among a list of distractors, which allows the drawbacks of different algorithms and sources of error to be studied. Experimental results show that our model can generate diverse, grammatically correct and content correlated questions that match the given answer.

Causality and Temporal Dependencies in the Design of Fault Management Systems

Reasoning about causes and effects naturally arises in the engineering of safety-critical systems. A classical example is Fault Tree Analysis, a deductive technique used for system safety assessment, whereby an undesired state is reduced to the set of its immediate causes. The design of fault management systems also requires reasoning on causality relationships. In particular, a fail-operational system needs to ensure timely detection and identification of faults, i.e. recognize the occurrence of run-time faults through their observable effects on the system. Even more complex scenarios arise when multiple faults are involved and may interact in subtle ways. In this work, we propose a formal approach to fault management for complex systems. We first introduce the notions of fault tree and minimal cut sets. We then present a formal framework for the specification and analysis of diagnosability, and for the design of fault detection and identification (FDI) components. Finally, we review recent advances in fault propagation analysis, based on the Timed Failure Propagation Graphs (TFPG) formalism.

Counterfactual Causality from First Principles?

In this position paper we discuss three main shortcomings of existing approaches to counterfactual causality from the computer science perspective, and sketch lines of work to try and overcome these issues: (1) causality definitions should be driven by a set of precisely specified requirements rather than specific examples; (2) causality frameworks should support system dynamics; (3) causality analysis should have a well-understood behavior in presence of abstraction.

A Decision Theoretic Approach to A/B Testing

A/B testing is ubiquitous within the machine learning and data science operations of internet companies. Generically, the idea is to perform a statistical test of the hypothesis that a new feature is better than the existing platform—for example, it results in higher revenue. If the p value for the test is below some pre-defined threshold—often, 0.05—the new feature is implemented. The difficulty of choosing an appropriate threshold has been noted before, particularly because dependent tests are often done sequentially, leading some to propose control of the false discovery rate (FDR) rather than use of a single, universal threshold. However, it is still necessary to make an arbitrary choice of the level at which to control FDR. Here we suggest a decision-theoretic approach to determining whether to adopt a new feature, which enables automated selection of an appropriate threshold. Our method has the basic ingredients of any decision-theory problem: a loss function, action space, and a notion of optimality, for which we choose Bayes risk. However, the loss function and the action space differ from the typical choices made in the literature, which has focused on the theory of point estimation. We give some basic results for Bayes-optimal thresholding rules for the feature adoption decision, and give some examples using eBay data. The results suggest that the 0.05 p-value threshold may be too conservative in some settings, but that its widespread use may reflect an ad-hoc means of controlling multiplicity in the common case of repeatedly testing variants of an experiment when the threshold is not reached.

Network of Recurrent Neural Networks

We describe a class of systems theory based neural networks called ‘Network Of Recurrent neural networks’ (NOR), which introduces a new structure level to RNN related models. In NOR, RNNs are viewed as the high-level neurons and are used to build the high-level layers. More specifically, we propose several methodologies to design different NOR topologies according to the theory of system evolution. Then we carry experiments on three different tasks to evaluate our implementations. Experimental results show our models outperform simple RNN remarkably under the same number of parameters, and sometimes achieve even better results than GRU and LSTM.

An Analysis of Dropout for Matrix Factorization

Dropout is a simple yet effective algorithm for regularizing neural networks by randomly dropping out units through Bernoulli multiplicative noise, and for some restricted problem classes, such as linear or logistic regression, several theoretical studies have demonstrated the equivalence between dropout and a fully deterministic optimization problem with data-dependent Tikhonov regularization. This work presents a theoretical analysis of dropout for matrix factorization, where Bernoulli random variables are used to drop a factor, thereby attempting to control the size of the factorization. While recent work has demonstrated the empirical effectiveness of dropout for matrix factorization, a theoretical understanding of the regularization properties of dropout in this context remains elusive. This work demonstrates the equivalence between dropout and a fully deterministic model for matrix factorization in which the factors are regularized by the sum of the product of the norms of the columns. While the resulting regularizer is closely related to a variational form of the nuclear norm, suggesting that dropout may limit the size of the factorization, we show that it is possible to trivially lower the objective value by doubling the size of the factorization. We show that this problem is caused by the use of a fixed dropout rate, which motivates the use of a rate that increases with the size of the factorization. Synthetic experiments validate our theoretical findings.

Fast and Strong Convergence of Online Learning Algorithms

In this paper, we study the online learning algorithm without explicit regularization terms. This algorithm is essentially a stochastic gradient descent scheme in a reproducing kernel Hilbert space (RKHS). The polynomially decaying step size in each iteration can play a role of regularization to ensure the generalization ability of online learning algorithm. We develop a novel capacity dependent analysis on the performance of the last iterate of online learning algorithm. The contribution of this paper is two-fold. First, our nice analysis can lead to the convergence rate in the standard mean square distance which is the best so far. Second, we establish, for the first time, the strong convergence of the last iterate with polynomially decaying step sizes in the RKHS norm. We demonstrate that the theoretical analysis established in this paper fully exploits the fine structure of the underlying RKHS, and thus can lead to sharp error estimates of online learning algorithm.

Mixed Precision Training

Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulates the gradients after each optimizer step. This single-precision copy is rounded to half-precision format during training. Secondly, we propose scaling the loss appropriately to handle the loss of information with half-precision gradients. We demonstrate that this approach works for a wide variety of models including convolution neural networks, recurrent neural networks and generative adversarial networks. This technique works for large scale models with more than 100 million parameters trained on large datasets. Using this approach, we can reduce the memory consumption of deep learning models by nearly 2x. In future processors, we can also expect a significant computation speedup using half-precision hardware units.

Almost-conserved operators in nearly many-body localized systems
Optimal Graphs for Independence and $k$-Independence Polynomials
Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition
Function space analysis of deep learning representation layers
$α$-Variational Inference with Statistical Guarantees
Considerations of automated machine learning in clinical metabolic profiling: Altered homocysteine plasma concentration associated with metformin exposure
Privacy-preserving Targeted Advertising
On decomposable random graphs
Multi-point distribution of periodic TASEP
Coreset based Dependency Networks
One-bit compressed sensing with partial Gaussian circulant matrices
Efficient mining of maximal biclusters in mixed-attribute datasets
Malliavin calculus approach to long exit times from an unstable equilibrium
The effect of prior probabilities on quantification and propagation of imprecise probabilities resulting from small datasets
Testing Independence between Observations from a Single Network of the Framingham Heart Study
Sum-Product Networks for Hybrid Domains
Clicks and Cliques. Exploring the Soul of the Community
Weighted allocations, their concomitant-based estimators, and asymptotics
On accurate domination in graphs
Blind Deconvolution by a Steepest Descent Algorithm on a Quotient Manifold
Tropicalization, symmetric polynomials, and complexity
Diffusion of a particle in the spatially correlated exponential random energy landscape: transition from normal to anomalous diffusion
Massive Open Online Courses Temporal Profiling for Dropout Prediction
Inspiration, Captivation, and Misdirection: Emergent Properties in Networks of Online Navigation
Standard detectors aren’t (currently) fooled by physical adversarial stop signs
On the Schur positivity of $Δ_{e_2} e_n[X]$
Iterative PET Image Reconstruction Using Convolutional Neural Network Representation
Geo-referencing Place from Everyday Natural Language Descriptions
What does Attention in Neural Machine Translation Pay Attention to?
The Role of Data-driven Priors in Multi-agent Crowd Trajectory Estimation
Balanced power diagrams for redistricting
Energy-efficient Amortized Inference with Cascaded Deep Classifiers
On the Service Capacity Region of Accessing Erasure Coded Content
Real-Time Action Detection in Video Surveillance using Sub-Action Descriptor with Multi-CNN
Critical ideals, minimum rank and zero forcing number
On Preemption and Overdetermination in Formal Theories of Causality
Efficient Dynamic Dictionary Matching with DAWGs and AC-automata
State Consensus for Discrete-time Multi-agent Systems over Time-varying Graphs
Prior Knowledge based mutation prioritization towards causal variant finding in rare disease
Structure of linear codes over the ring $B_k$
AdaDNNs: Adaptive Ensemble of Deep Neural Networks for Scene Text Recognition
Learning to Rank Question-Answer Pairs using Hierarchical Recurrent Encoder with Latent Topic Clustering
Combinatorial and Asymptotical Results on the Neighborhood Grid
Statistical Methods and Workflow for Analyzing Human Metabolomics Data
BestConfig: Tapping the Performance Potential of Systems via Automatic Configuration Tuning
On- and Off-Policy Monotonic Policy Improvement
Quantitative Comparison of Statistical Methods for Analyzing Human Metabolomics Data
Safe Semi-Supervised Learning of Sum-Product Networks
Weak localization of magnons in a disordered two-dimensional antiferromagnet
Negative magneto-thermal-resistance in a disordered two-dimensional antiferromagnet
Functions to map photoelectron distributions in a variety of setups in angle-resolved photoemission spectroscopy
The Sparse Multivariate Method of Simulated Quantiles
Balanced Truncation of Networked Linear Passive Systems
Reversal property of the Brownian tree
Learning to Generalize: Meta-Learning for Domain Generalization
DocEmul: a Toolkit to Generate Structured Historical Documents
Sparse Semidefinite Programs with Near-Linear Time Complexity
MoNoise: Modeling Noise Using a Modular Normalization System
A Note on Nesting in Dyadic Deontic Logic
Automatic Streaming Segmentation of Stereo Video Using Bilateral Space
An optimised multi-arm multi-stage clinical trial design for unknown variance
Group Sequential Crossover Trial Designs with Strong Control of the Familywise Error Rate
Yet another skew-elliptical family but of a different kind: return to Lemma 1
Admissible multi-arm stepped-wedge cluster randomized trial designs
A Very Low Resource Language Speech Corpus for Computational Language Documentation Experiments
A buffer Hawkes process for limit order books
Volatility estimation for stochastic PDEs using high-frequency observations
Underestimated cost of targeted attacks on complex networks
The Overfull Conjecture on Split-Comparability Graphs
Measuring the gradualist approach to internationalization
On Louchard’s Asymptotic Series
On Dimension-Free Linear System
Contaminated speech training methods for robust DNN-HMM distant speech recognition
Mutual information between reflected and transmitted speckle images
Improved nonparametric estimation of the drift in diffusion processes
Exact integrated completed likelihood maximisation in a stochastic block transition model for dynamic networks
Traffic Sign Timely Visual Recognizability Evaluation Based on 3D Measurable Point Clouds
On diffusions in media with pockets of large diffusivity
321-avoiding affine permutations and their many heaps
A general version of Price’s theorem
A Generalized Algebraic Approach to Optimizing SC-LDPC Codes
Regularization effects of a noise propagating through a chain of differential equations: an almost sharp result
Near-misses in Wilf’s conjecture
An extension of the Polyak convexity principle with application to nonconvex optimization
Base Station Diversity Propagation Measurements at 73 GHz Millimeter-Wave for 5G Coordinated Multipoint (CoMP) Analysis
Multilevel Modeling with Structured Penalties for Classification from Imaging Genetics data
Angular Accuracy of Steerable Feature Detectors
LinXGBoost: Extension of XGBoost to Generalized Local Linear Models
Evaluation of Harmonic Sums with Integrals
Beam Management for Millimeter Wave Beamspace MU-MIMO Systems
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
Optimized Frameless ALOHA for Cooperative Base Stations with Overlapped Coverage Areas
Accelerating Energy Games Solvers on Modern Architectures
Decentralized Resource Discovery and Management for Future Manycore Systems
Colored discrete spaces: higher dimensional combinatorial maps and quantum gravity
High-dimensional dynamics of generalization error in neural networks
Identifying Active Sites of the Water-Gas Shift Reaction over Titania Supported Platinum Catalysts under Uncertainty
Underestimate Sequences via Quadratic Averaging
Representations and evaluation strategies for feasibly approximable functions
Motor Insurance Accidental Damage Claims Modeling with Factor Collapsing and Bayesian Model Averaging
Linear response for random dynamical systems
LaSalle Invariance Principle for Discrete-time Dynamical Systems: A Concise and Self-contained Tutorial
RLT2-based Parallel Algorithms for Solving Large Quadratic Assignment Problems on Graphics Processing Unit Clusters
Saturation of Berge Hypergraphs
Confidence through Attention
Erdos-Hajnal conjecture for graphs with bounded VC-dimension
Bayesian Attitude Estimation with the Matrix Fisher Distribution on SO(3)
Emergent Complexity via Multi-Agent Competition