MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for scaling Deep Learning

Existing Deep Learning frameworks exclusively use either Parameter Server(PS) approach or MPI parallelism. In this paper, we discuss the drawbacks of such approaches and propose a generic framework supporting both PS and MPI programming paradigms, co-existing at the same time. The key advantage of the new model is to embed the scaling benefits of MPI parallelism into the loosely coupled PS task model. Apart from providing a practical usage model of MPI in cloud, such framework allows for novel communication avoiding algorithms that do parameter averaging in Stochastic Gradient Descent(SGD) approaches. We show how MPI and PS models can synergestically apply algorithms such as Elastic SGD to improve the rate of convergence against existing approaches. These new algorithms directly help scaling SGD clusterwide. Further, we also optimize the critical component of the framework, namely global aggregation or allreduce using a novel concept of tensor collectives. These treat a group of vectors on a node as a single object allowing for the existing single vector algorithms to be directly applicable. We back our claims with sufficient emperical evidence using large scale ImageNet 1K data. Our framework is built upon MXNET but the design is generic and can be adapted to other popular DL infrastructures.

BioWorkbench: A High-Performance Framework for Managing and Analyzing Bioinformatics Experiments

Advances in sequencing techniques have led to exponential growth in biological data, demanding the development of large-scale bioinformatics experiments. Because these experiments are computation- and data-intensive, they require high-performance computing (HPC) techniques and can benefit from specialized technologies such as Scientific Workflow Management Systems (SWfMS) and databases. In this work, we present BioWorkbench, a framework for managing and analyzing bioinformatics experiments. This framework automatically collects provenance data, including both performance data from workflow execution and data from the scientific domain of the workflow application. Provenance data can be analyzed through a web application that abstracts a set of queries to the provenance database, simplifying access to provenance information. We evaluate BioWorkbench using three case studies: SwiftPhylo, a phylogenetic tree assembly workflow; SwiftGECKO, a comparative genomics workflow; and RASflow, a RASopathy analysis workflow. We analyze each workflow from both computational and scientific domain perspectives, by using queries to a provenance and annotation database. Some of these queries are available as a pre-built feature of the BioWorkbench web application. Through the provenance data, we show that the framework is scalable and achieves high-performance, reducing up to 98% of the case studies execution time. We also show how the application of machine learning techniques can enrich the analysis process.

On Multivariate Optimal Transportation

The Wasserstein distance, under a convex cost function, between random variables on \mathbb{R} admits a simple, tractable solution. On the other hand, in higher dimensions, and even for the quadratic cost, the distance and associated optimal coupling, remains problematic, save for a few special cases. In this paper, we demonstrate that, in the case of quadratic cost, it is the underlying copula which drives optimal couplings. That is, if one has the optimal coupling between the two copulas of two multivariate random variables, then one has it between the two multivariate random variables. This motivates looking at optimal couplings between copula and indeed we find such for the class of Archimedean copula. The technique we use to do this is then developed to find optimal couplings between copulas associated with elliptical distributions.

Neural Program Synthesis with Priority Queue Training

We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to find programs with maximal rewards. We employ an iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far. Then, we synthesize new programs and add them to the priority queue by sampling from the RNN. We benchmark our algorithm, called priority queue training (or PQT), against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF. Our experimental results show that our simple PQT algorithm significantly outperforms the baselines. By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs.

Unsupervised Part-of-Speech Induction

Part-of-Speech (POS) tagging is an old and fundamental task in natural language processing. While supervised POS taggers have shown promising accuracy, it is not always feasible to use supervised methods due to lack of labeled data. In this project, we attempt to unsurprisingly induce POS tags by iteratively looking for a recurring pattern of words through a hierarchical agglomerative clustering process. Our approach shows promising results when compared to the tagging results of the state-of-the-art unsupervised POS taggers.

Graphical Models for Processing Missing Data

This paper reviews recent advances in missing data research using graphical models to represent multivariate dependencies. We first examine the limitations of traditional frameworks from three different perspectives: \textit{transparency, estimability and testability}. We then show how procedures based on graphical models can overcome these limitations and provide meaningful performance guarantees even when data are Missing Not At Random (MNAR). In particular, we identify conditions that guarantee consistent estimation in broad categories of missing data problems, and derive procedures for implementing this estimation. Finally we derive testable implications for missing data models in both MAR (Missing At Random) and MNAR categories.

Using probabilistic programs as proposals

Monte Carlo inference has asymptotic guarantees, but can be slow when using generic proposals. Handcrafted proposals that rely on user knowledge about the posterior distribution can be efficient, but are difficult to derive and implement. This paper proposes to let users express their posterior knowledge in the form of \emph{proposal programs}, which are samplers written in probabilistic programming languages. One strategy for writing good proposal programs is to combine domain-specific heuristic algorithms with neural network models. The heuristics identify high probability regions, and the neural networks model the posterior uncertainty around the outputs of the algorithm. Proposal programs can be used as proposal distributions in importance sampling and Metropolis-Hastings samplers without sacrificing asymptotic consistency, and can be optimized offline using inference compilation. Support for optimizing and using proposal programs is easily implemented in a sampling-based probabilistic programming runtime. The paper illustrates the proposed technique with a proposal program that combines RANSAC and neural networks to accelerate inference in a Bayesian linear regression with outliers model.

On Evaluating and Comparing Conversational Agents

Conversational agents are exploding in popularity. However, much work remains in the area of non goal-oriented conversations, despite significant growth in research interest over recent years. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar university competition where sixteen selected university teams built conversational agents to deliver the best social conversational experience. Alexa Prize provided the academic community with the unique opportunity to perform research with a live system used by millions of users. The subjectivity associated with evaluating conversations is key element underlying the challenge of building non-goal oriented dialogue systems. In this paper, we propose a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human judgement. The proposed metrics provide granular analysis of the conversational agents, which is not captured in human ratings. We show that these metrics can be used as a reasonable proxy for human judgment. We provide a mechanism to unify the metrics for selecting the top performing agents, which has also been applied throughout the Alexa Prize competition. To our knowledge, to date it is the largest setting for evaluating agents with millions of conversations and hundreds of thousands of ratings from users. We believe that this work is a step towards an automatic evaluation process for conversational AIs.

Sharp instruments for classifying compliers and generalizing causal effects

It is well-known that, without restricting treatment effect heterogeneity, instrumental variable (IV) methods only identify ‘local’ effects among compliers, i.e., those subjects who take treatment only when encouraged by the IV. Local effects are controversial since they seem to only apply to an unidentified subgroup; this has led many to denounce these effects as having little policy relevance. However, we show that such pessimism is not always warranted: it is possible in some cases to accurately predict who compliers are, and obtain tight bounds on more generalizable effects in identifiable subgroups. We propose methods for doing so and study their estimation error and asymptotic properties, showing that these tasks can in theory be accomplished even with very weak IVs. We go on to introduce a new measure of IV quality called ‘sharpness’, which reflects the variation in compliance explained by covariates, and captures how well one can identify compliers and obtain tight bounds on identifiable subgroup effects. We develop an estimator of sharpness, and show that it is asymptotically efficient under weak conditions. Finally we explore finite-sample properties via simulation, and apply the methods to study canvassing effects on voter turnout. We propose that sharpness should be presented alongside strength to assess IV quality.

Quantization/clustering: when does k-means work?

Though mostly used as a clustering algorithm, k-means are originally designed as a quantization algorithm. Namely, it aims at providing a compression of a probability distribution with k points. Building upon [21, 33], we try to investigate how and when these two approaches are compatible. Namely, we show that provided the sample distribution satisfies a margin like condition (in the sense of [27] for supervised learning), both the associated empirical risk minimizer and the output of Lloyd’s algorithm provide almost optimal classification in certain cases (in the sense of [6]). Besides, we also show that they achieved fast and optimal convergence rates in terms of sample size and compression risk.

Soft Locality Preserving Map (SLPM) for Facial Expression Recognition

For image recognition, an extensive number of methods have been proposed to overcome the high-dimensionality problem of feature vectors being used. These methods vary from unsupervised to supervised, and from statistics to graph-theory based. In this paper, the most popular and the state-of-the-art methods for dimensionality reduction are firstly reviewed, and then a new and more efficient manifold-learning method, named Soft Locality Preserving Map (SLPM), is presented. Furthermore, feature generation and sample selection are proposed to achieve better manifold learning. SLPM is a graph-based subspace-learning method, with the use of k-neighbourhood information and the class information. The key feature of SLPM is that it aims to control the level of spread of the different classes, because the spread of the classes in the underlying manifold is closely connected to the generalizability of the learned subspace. Our proposed manifold-learning method can be applied to various pattern recognition applications, and we evaluate its performances on facial expression recognition. Experiments on databases, such as the Bahcesehir University Multilingual Affective Face Database (BAUM-2), the Extended Cohn-Kanade (CK+) Database, the Japanese Female Facial Expression (JAFFE) Database, and the Taiwanese Facial Expression Image Database (TFEID), show that SLPM can effectively reduce the dimensionality of the feature vectors and enhance the discriminative power of the extracted features for expression recognition. Furthermore, the proposed feature-generation method can improve the generalizability of the underlying manifolds for facial expression recognition.

EARL: Joint Entity and Relation Linking for Question Answering over Knowledge Graphs

In order to answer natural language questions over knowledge graphs, most processing pipelines involve entity and relation linking. Traditionally, entity linking and relation linking has been performed either as dependent sequential tasks or independent parallel tasks. In this paper, we propose a framework, called EARL, which performs entity linking and relation linking as a joint single task. EARL is modeled on an optimised variation of GeneralisedTravelling Salesperson Problem. The system determines the best semantic connection between all keywords of the question by referring to the knowledge graph. This is achieved by exploiting the connection density between entity candidates and relation candidates. We have empirically evaluated the framework on a dataset with 3000 complex questions. Our system surpasses state-of-the-art scores for entity linking task by reporting an accuracy of 0.67against 0.40 from the next best entity linker

Robust inference with knockoffs

We consider the variable selection problem, which seeks to identify important variables influencing a response Y out of many candidate features X_1, \ldots, X_p. We wish to do so while offering finite-sample guarantees about the fraction of false positives – selected variables X_j that in fact have no effect on Y after the other features are known. When the number of features p is large (perhaps even larger than the sample size n), and we have no prior knowledge regarding the type of dependence between Y and X, the model-X knockoffs framework nonetheless allows us to select a model with a guaranteed bound on the false discovery rate, as long as the distribution of the feature vector X=(X_1,\dots,X_p) is exactly known. This model selection procedure operates by constructing ‘knockoff copies” of each of the p features, which are then used as a control group to ensure that the model selection algorithm is not choosing too many irrelevant features. In this work, we study the practical setting where the distribution of X could only be estimated, rather than known exactly, and the knockoff copies of the X_j‘s are therefore constructed somewhat incorrectly. Our results, which are free of any modeling assumption whatsoever, show that the resulting model selection procedure incurs an inflation of the false discovery rate that is proportional to our errors in estimating the distribution of each feature X_j conditional on the remaining features \{X_k:k\neq j\}. The model-X knockoff framework is therefore robust to errors in the underlying assumptions on the distribution of X, making it an effective method for many practical applications, such as genome-wide association studies, where the underlying distribution on the features X_1,\dots,X_p is estimated accurately but not known exactly.

Stochastic Learning of Nonstationary Kernels for Natural Language Modeling

Natural language processing often involves computations with semantic or syntactic graphs to facilitate sophisticated reasoning based on structural relationships. While convolution kernels provide a powerful tool for comparing graph structure based on node (word) level relationships, they are difficult to customize and can be computationally expensive. We propose a generalization of convolution kernels, with a nonstationary model, for better expressibility of natural languages in supervised settings. For a scalable learning of the parameters introduced with our model, we propose a novel algorithm that leverages stochastic sampling on k-nearest neighbor graphs, along with approximations based on locality-sensitive hashing. We demonstrate the advantages of our approach on a challenging real-world (structured inference) problem of automatically extracting biological models from the text of scientific papers.

Black Holes as Brains: Neural Networks with Area Law Entropy

Motivated by the potential similarities between the underlying mechanisms of the enhanced memory storage capacity in black holes and in brain networks, we construct an artificial quantum neural network based on gravity-like synaptic connections and a symmetry structure that allows to describe the network in terms of geometry of a d-dimensional space. We show that the network possesses a critical state in which the gapless neurons emerge that appear to inhabit a (d-1)-dimensional surface, with their number given by the surface area. In the excitations of these neurons, the network can store and retrieve an exponentially large number of patterns within an arbitrarily narrow energy gap. The corresponding micro-state entropy of the brain network exhibits an area law. The neural network can be described in terms of a quantum field, via identifying the different neurons with the different momentum modes of the field, while identifying the synaptic connections among the neurons with the interactions among the corresponding momentum modes. Such a mapping allows to attribute a well-defined sense of geometry to an intrinsically non-local system, such as the neural network, and vice versa, it allows to represent the quantum field model as a neural network.

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called ‘perceptual losses’? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

Proceedings of the Workshop on High Performance Energy Efficient Embedded Systems (HIP3ES) 2018
Communication-Constrained Expansion Planning for Resilient Distribution Systems
Strong Sure Screening of Ultra-high Dimensional Categorical Data
Oddities of quantum colorings
Segment-based Methods for Facial Attribute Detection from Partial Faces
On strong $L^2$ convergence of numerical schemes for the stochastic 2D Navier-Stokes equations
From Superpixel to Human Shape Modelling for Carried Object Detection
On the Noise-Information Separation of a Private Principal Component Analysis Scheme
Rate Selection and Power Adaptation using Maximal Ratio Combining for the Random Access Gaussian Channel
Inference Suboptimality in Variational Autoencoders
SemiCompRisks: An R Package for Independent and Cluster-Correlated Analyses of Semi-Competing Risks Data
Spiking spin-glass models for label propagation and community detection
Learning Aided Optimization for Energy Harvesting Devices with Outdated State Information
The Structure of the Three-Dimensional Special Linear Group over a Local Field
DuctTeip: An efficient programming model for distributed task based parallel computing
Deterministic search for CNF satisfying assignments in almost polynomial time
Task parallel implementation of a solver for electromagnetic scattering problems
Improved pseudorandom generators from pseudorandom multi-switching lemmas
Characterization and Approximation of Strong General Dual Feasible Functions
Estimation of the Robin coefficient field in a Poisson problem with uncertain conductivity field
Finite Blocklength and Dispersion Bounds for the Arbitrarily-Varying Channel
Local Map-assisted Positioning for Flying Wireless Relays
A framework for measuring dependence between random vectors
Multi-Level Stochastic Gradient Methods for Nested Compositon Optimization
Closed formulas for exponential sums of symmetric polynomials over Galois fields
SEE: Syntax-aware Entity Embedding for Neural Relation Extraction
Conversational AI: The Science Behind the Alexa Prize
Quadrature compressive sampling SAR imaging
Deep Classification of Epileptic Signals
Improved English to Russian Translation by Neural Suffix Prediction
Parity-Check Polar Coding for 5G and Beyond
RAF: Robust Adaptive Multi-Feedback Channel Estimation for Millimeter Wave MIMO Systems
Topic-based Evaluation for Conversational Bots
Optimal locally repairable codes of distance $3$ and $4$ via cyclic codes
Applying Vector Space Model (VSM) Techniques in Information Retrieval for Arabic Language
Restless Bandits for Dynamic Sourcing of Resources in Social and Information Networks
A current value Hamiltonian Approach for Discrete time Optimal Control Problems arising in Economic Growth
To Relay or not to Relay: Open Distance and Optimal Deployment for Linear Underwater Acoustic Networks
Multidimensional Range Queries on Modern Hardware
A tool framework for tweaking features in synthetic datasets
Can Negligible Cooperation Increase Network Capacity? The Average-Error Case
Generalized Quantum Shannon Impossibility for Quantum Encryption
From Uncertainty Data to Robust Policies for Temporal Logic Planning
Asynchronous Mobile-Edge Computation Offloading: Energy-Efficient Resource Management
Diffusion limits for a Markov modulated counting process
On the Reliability Function of Distributed Hypothesis Testing Under Optimal Detection
Repairing the Faure-Loidreau Public-Key Cryptosystem
Universal random codes: Capacity regions of the compound quantum multiple-access channel with one classical and one quantum sender
Exact Calculation of Normalized Maximum Likelihood Code Length Using Fourier Analysis
Polypus: a Big Data Self-Deployable Architecture for Microblogging Text Extraction and Real-Time Sentiment Analysis
Multi-Band Covariance Interpolation with Applications in Massive MIMO
How to Split UL/DL Antennas in Full-Duplex Cellular Networks
Viable Insider Markets
Large deviation theory for diluted Wishart random matrices
Enhancing Physical Layer Security in AF Relay Assisted Multi-Carrier Wireless Transmission
PALE: Partially Asynchronous Agile Leader Election
Counterfactual equivalence for POMDPs, and underlying deterministic environments
Double asymptotic for random walks on hypercubes
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
Performance Evaluation of Advanced Relaying Protocols in Large Wireless Networks
Improved asynchronous parallel optimization analysis for stochastic incremental methods
Estimation of Local Anisotropy Based on Level Sets
Non-stationary Douglas-Rachford and alternating direction method of multipliers: adaptive stepsizes and convergence
A Sundaram type bijection for $\mathrm{SO}(3)$: vacillating tableaux and pairs of standard Young tableaux and orthogonal Littlewood-Richardson tableaux
On the precision matrix of an irregularly sampled AR(1) process
A Hybrid NOMA-TDMA for Multiple Access Channels with Non-Ideal Batteries and Circuit Cost
Cortical-inspired image reconstruction via sub-Riemannian geometry and hypoelliptic diffusion
Energy Harvesting Communications Using Dual Alternating Batteries
Two-valued local sets of the 2D continuum Gaussian free field: connectivity, labels, and induced metrics
Convergence Analysis of Sample Average Approximation of Two-state Stochastic Generalized Equations
The probabilities of trees and cladograms under Ford’s $α$-model
Enhancing Translation Language Models with Word Embedding for Information Retrieval
Autoencoders and Probabilistic Inference with Missing Data: An Exact Solution for The Factor Analysis Case
Oink: an Implementation and Evaluation of Modern Parity Game Solvers
Symmetric road interchanges
Construction and Performance of Quantum Burst Error Correction Codes for Correlated Errors
A Monetary Mechanism for Stabilizing Cooperative Data Exchange with Selfish Users
Hopf algebras on decorated noncrossing arc diagrams
An entropy inequality for symmetric random variables
From Azéma supermartingales of finite honest times to optional semimartingales of class-($Σ$)
Paths to Understanding Birational Rowmotion on Products of Two Chains
Parameterized (Approximate) Defective Coloring
Zeroth-order general Randic index of $k$-generalized quasi trees
Privacy in Index Coding: Improved Bounds and Coding Schemes
On Locally Decodable Index Codes
A parallel workload has extreme variability in a production environment
Connecting the Kuramoto Model and the Chimera State
Modeling High-Dimensional Data with Case-Control Sampling and Dependency Structures
Learning and Inferring a Driver’s Braking Action in Car-Following Scenarios
Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction
Phase transitions in social networks inspired by the Schelling model
Inner Bound for the Capacity Region of Noisy Channels with an Authentication Requirement