Constructing Deep Neural Networks by Bayesian Network Structure Learning

We introduce a principled approach for unsupervised structure learning of deep neural networks. We propose a new interpretation for depth and inter-layer connectivity where conditional independencies in the input distribution are encoded hierarchically in the network structure. Thus, the depth of the network is determined inherently (equal to the maximal order of independence in the input distribution). The proposed method casts the problem of neural network structure learning as a problem of Bayesian network structure learning. Then, instead of directly learning the discriminative structure, it learns a generative graph, constructs its stochastic inverse, and then constructs a discriminative graph. We prove that conditional-dependency relations among the latent variables in the generative graph are preserved in the class-conditional discriminative graph. We demonstrate on image classification benchmarks that the deepest layers (convolutional and dense) of common networks can be replaced by significantly smaller learned structures, while maintaining classification accuracy—state-of-the-art on tested benchmarks. Our structure learning algorithm requires a small computational cost and runs efficiently on a standard desktop CPU.

SSIMLayer: Towards Robust Deep Representation Learning via Nonlinear Structural Similarity

Deeper convolutional neural networks provide more capacity to approximate complex mapping functions. However, increasing network depth imposes difficulties on training and increases model complexity. This paper presents a new nonlinear computational layer of considerably high capacity to the deep convolutional neural network architectures. This layer performs a set of comprehensive convolution operations that mimics the overall function of the human visual system (HVS) via focusing on learning structural information in its input. The core of its computations is evaluating the components of the structural similarity metric (SSIM) in a setting that allows the kernels to learn to match structural information. The proposed SSIMLayer is inherently nonlinear and hence, it does not require subsequent nonlinear transformations. Experiments conducted on CIFAR-10 benchmark demonstrates that the SSIMLayer provides better convergence than the traditional convolutional layer, bypasses the need for nonlinear transformations and shows more robustness against noise perturbations and adversarial attacks.

The Internet of Things: Secure Distributed Inference

The growth in the number of devices connected to the Internet of Things (IoT) poses major challenges in security. The integrity and trustworthiness of data and data analytics are increasingly important concerns in IoT applications. These are compounded by the highly distributed nature of IoT devices, making it infeasible to prevent attacks and intrusions on all data sources. Adversaries may hijack devices and compromise their data. As a result, reactive countermeasures, such as intrusion detection and resilient analytics, become vital components of security. This paper overviews algorithms for secure distributed inference in IoT.

Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions

The current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the computation and parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, the high energy consumption of convolutions limits its deployment on mobile devices. To this end, we proposed a simple yet effective scheme for compressing convolutions though applying k-means clustering on the weights, compression is achieved through weight-sharing, by only recording K cluster centers and weight assignment indexes. We then introduced a novel spectrally relaxed k-means regularization, which tends to make hard assignments of convolutional layer weights to K learned cluster centers during re-training. We additionally propose an improved set of metrics to estimate energy consumption of CNN hardware implementations, whose estimation results are verified to be consistent with previously proposed energy estimation tool extrapolated from actual hardware measurements. We finally evaluated Deep k-Means across several CNN models in terms of both compression ratio and energy consumption reduction, observing promising results without incurring accuracy loss. The code is available at https://…/Deep-K-Means

Evaluation of Information Retrieval Systems Using Structural Equation Modelling

The interpretation of the experimental data collected by testing systems across input datasets and model parameters is of strategic importance for system design and implementation. In particular, finding relationships between variables and detecting the latent variables affecting retrieval performance can provide designers, engineers and experimenters with useful if not necessary information about how a system is performing. This paper discusses the use of Structural Equation Modelling (SEM) in providing an in-depth explanation of evaluation results and an explanation of failures and successes of a system; in particular, we focus on the case of Information Retrieval.

SAQL: A Stream-based Query System for Real-Time Abnormal System Behavior Detection

Recently, advanced cyber attacks, which consist of a sequence of steps that involve many vulnerabilities and hosts, compromise the security of many well-protected businesses. This has led to the solutions that ubiquitously monitor system activities in each host (big data) as a series of events, and search for anomalies (abnormal behaviors) for triaging risky events. Since fighting against these attacks is a time-critical mission to prevent further damage, these solutions face challenges in incorporating expert knowledge to perform timely anomaly detection over the large-scale provenance data. To address these challenges, we propose a novel stream-based query system that takes as input, a real-time event feed aggregated from multiple hosts in an enterprise, and provides an anomaly query engine that queries the event feed to identify abnormal behaviors based on the specified anomalies. To facilitate the task of expressing anomalies based on expert knowledge, our system provides a domain-specific query language, SAQL, which allows analysts to express models for (1) rule-based anomalies, (2) time-series anomalies, (3) invariant-based anomalies, and (4) outlier-based anomalies. We deployed our system in NEC Labs America comprising 150 hosts and evaluated it using 1.1TB of real system monitoring data (containing 3.3 billion events). Our evaluations on a broad set of attack behaviors and micro-benchmarks show that our system has a low detection latency (<2s) and a high system throughput (110,000 events/s; supporting ~4000 hosts), and is more efficient in memory utilization than the existing stream-based complex event processing systems.

A Tour of Reinforcement Learning: The View from Continuous Control

This manuscript surveys reinforcement learning from the perspective of optimization and control with a focus on continuous control applications. It surveys the general formulation, terminology, and typical experimental implementations of reinforcement learning and reviews competing solution paradigms. In order to compare the relative merits of various techniques, this survey presents a case study of the Linear Quadratic Regulator (LQR) with unknown dynamics, perhaps the simplest and best studied problem in optimal control. The manuscript describes how merging techniques from learning theory and control can provide non-asymptotic characterizations of LQR performance and shows that these characterizations tend to match experimental behavior. In turn, when revisiting more complex applications, many of the observed phenomena in LQR persist. In particular, theory and experiment demonstrate the role and importance of models and the cost of generality in reinforcement learning algorithms. This survey concludes with a discussion of some of the challenges in designing learning systems that safely and reliably interact with complex and uncertain environments and how tools from reinforcement learning and controls might be combined to approach these challenges.

Inference Trees: Adaptive Inference with Exploration

We introduce inference trees (ITs), a new class of inference methods that build on ideas from Monte Carlo tree search to perform adaptive sampling in a manner that balances exploration with exploitation, ensures consistency, and alleviates pathologies in existing adaptive methods. ITs adaptively sample from hierarchical partitions of the parameter space, while simultaneously learning these partitions in an online manner. This enables ITs to not only identify regions of high posterior mass, but also maintain uncertainty estimates to track regions where significant posterior mass may have been missed. ITs can be based on any inference method that provides a consistent estimate of the marginal likelihood. They are particularly effective when combined with sequential Monte Carlo, where they capture long-range dependencies and yield improvements beyond proposal adaptation alone.

On $r$-Simple $k$-Path and Related Problems Parameterized by $k/r$
Exact correlations in the nonequilibrium stationary state of the noisy Kuramoto model
A Design of FPGA Based Small Animal PET Real Time Digital Signal Processing and Correction Logic
A New All-Digital Background Calibration Technique for Time-Interleaved ADC Using First Order Approximation FIR Filters
Channel Estimation for Massive MIMO Communication System Using Deep Neural Network
On the protocol dependence of plasticity in ultra-stable amorphous solids
Generating functions for multiple zeta star values
On Energy-Efficient NOMA Designs for Heterogeneous Low-Latency Downlink Transmissions
Urn Models and Fibonacci Series
Self-Organized Criticality and Pattern Emergence through the lens of Tropical Geometry
Inferring Routing Preferences of Bicyclists from Sparse Sets of Trajectories
Fusion of complex networks and randomized neural networks for texture analysis
Dilated Temporal Fully-Convolutional Network for Semantic Segmentation of Motion Capture Data
Some combinatorial identities appearing in the calculation of the cohomology of Siegel modular varieties
A Unified Analysis of Random Fourier Features
The analytic rank of tensors is subadditive, and its applications
Convergence analysis of a cell centered finite volume diffusion operator on non-orthogonal polyhedral meshes
A Deeper Look at Power Normalizations
On Nondeterministic Derandomization of Freivalds’ Algorithm: Consequences, Avenues and Algorithmic Progress
On The Differential Privacy of Thompson Sampling With Gaussian Prior
Balanced News Using Constrained Bandit-based Personalization
N-Gram Graph, A Novel Molecule Representation
Equalizing Financial Impact in Supervised Learning
Distributionally Robust Optimization with Decision Dependent Ambiguity Sets
Optimal periodic replenishment policies for spectrally positive Lévy demand processes
Analysis of Krylov Subspace Solutions of Regularized Nonconvex Quadratic Problems
Cyber-Physical Specification Mismatches
Scale Space Approximation in Convolutional Neural Networks for Retinal Vessel Segmentation
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network
JR-GAN: Jacobian Regularization for Generative Adversarial Networks
Two Deletion Correcting Codes from Indicator Vectors
FBI-Pose: Towards Bridging the Gap between 2D Images and 3D Human Poses using Forward-or-Backward Information
A Scalable Machine Learning System for Pre-Season Agriculture Yield Forecast
Energy-Efficient Extended Sub-connected Architecture for Hybrid Precoding in Millimeter Massive Wave MIMO Systems
Additive phase-noise in frequency conversion in LLRF systems
PILOT: A Pixel Intensity Driven Illuminant Color Estimation Framework for Color Constancy
T0 Fan-out for Back-n White Neutron Facility at CSNS
Electronics of Time-of-flight Measurement for Back-n at CSNS
Optimal Online Contention Resolution Schemes via Ex-Ante Prophet Inequalities
Effect of Transit Signal Priority on Bus Service Reliability
Track Xplorer: A System for Visual Analysis of Sensor-based Motor Activity Predictions
Development of the Front-End Electronics for PandaX-III Prototype TPC
Real-time Data Flow Control for CBM-TOF Super Module Quality Evaluation
Bené: On Demand Cost-Effective Scaling at the Edge
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision
Cavity Simulator for European Spallation Source
Noise Measurements of High-Speed, Light-Emitting GaN Resonant-Tunneling Diodes
Towards Optimal Transport with Global Invariances
Best Vision Technologies Submission to ActivityNet Challenge 2018-Task: Dense-Captioning Events in Videos
Framework for Opinion Mining Approach to Augment Education System Performance
Real-Time Redundancy for the 1.3 GHz Master Oscillator of the European-XFEL
RAM: A Region-Aware Deep Model for Vehicle Re-Identification
A new benchmark set for Traveling salesman problem and Hamiltonian cycle problem
Ohno type relations for classical and finite multiple zeta-star values
Improving Chemical Autoencoder Latent Space and Molecular De novo Generation Diversity with Heteroencoders
Exit problem as the generalized solution of Dirichlet problem
Semiparametrically Point-Optimal Hybrid Rank Tests for Unit Roots
Vision-based Pose Estimation for Augmented Reality : A Comparison Study
Kick control: using the attracting states arising within the sensorimotor loop of self-organized robots as motor primitives
Single-channel Speech Dereverberation via Generative Adversarial Training
Outage Analysis of Relay-Assisted mmWave Cellular Systems Employing JSDM
Diversified Late Acceptance Search
Convergence of transport noise to Ornstein-Uhlenbeck for 2D Euler equations under the enstrophy measure
Semi-intrusive Uncertainty Quantification for Multiscale models
Sparse 3D Point-cloud Map Upsampling and Noise Removal as a vSLAM Post-processing Step: Experimental Evaluation
Partial least squares discriminant analysis: A dimensionality reduction method to classify hyperspectral data
Finitary isomorphisms of some infinite entropy Bernoulli flows
Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewards
The Tutte’s condition in terms of graph factors
Shifted critical threshold in the loop ${ \boldsymbol{O(n)}}$ model at arbitrary small $\boldsymbol{n}$
Approximate Bayesian inference for mixture cure models
Distance covariance for discretized stochastic processes
Fast ASR-free and almost zero-resource keyword spotting using DTW and CNNs for humanitarian monitoring
Blowups and blowdowns of geodesics in Carnot groups
Wireless Power Transfer in Cooperative DF Relaying Networks with Log-Normal Fading
A Design Space Exploration (DSE) on Non-Invasive Sensing of Bladder Filling Using Near Infrared Spectroscopy (NIRS)
Resolution with Counting: Lower Bounds over Different Moduli
An Unsupervised Learning Classifier with Competitive Error Performance
Generalized additive models for location, scale and shape for program evaluation: A guide to practice
Outage of Periodic Downlink Wireless Networks with Hard Deadlines
Accelerating likelihood optimization for ICA on real signals
Reliable Transmission of Short Packets through Queues and Noisy Channels under Latency and Peak-Age Violation Guarantees
Quasi-likelihood analysis of an ergodic diffusion plus noise
Exponential weights in multivariate regression and a low-rankness favoring prior
Transmission-Constrained Unit Commitment
Exploring Adversarial Examples: Patterns of One-Pixel Attacks
Even Longer Cycles in Essentially 4-Connected Planar Graphs
Structural and Topological Nature of Plasticity in Sheared Granular Materials
Beamforming Design and Power Allocation for Secure Transmission with NOMA
Sum-of-Squares meets Nash: Optimal Lower Bounds for Finding any Equilibrium
A Distributed Flexible Delay-tolerant Proximal Gradient Algorithm
Estimating Lower Probability Bound of Power System’s Capability to Fully Accommodate Variable Wind Generation
Propagating Uncertainty through the tanh Function with Application to Reservoir Computing
Predicting Effective Control Parameters for Differential Evolution using Cluster Analysis of Objective Function Features
Prior Attention for Style-aware Sequence-to-Sequence Models
Gaussian process regression for forest attribute estimation from airborne laser scanning data
Encoding shortest paths in graphs assuming the code is queried using bit-wise comparison
A Transferable Pedestrian Motion Prediction Model for Intersections with Different Geometries
A Unified Model with Structured Output for Fashion Images Classification
Handling Massive N-Gram Datasets Efficiently
Context-Aware Pedestrian Motion Prediction In Urban Intersections
Compact Policies for Fully-Observable Non-Deterministic Planning as SAT
Does data interpolation contradict statistical optimality
Accounting for phenology in the analysis of animal movement
Optimal stopping of McKean-Vlasov diffusions via regression on particle systems
Finding Optimal Solutions to Token Swapping by Conflict-based Search and Reduction to SAT
Optimal control of diffusion processes pertaining to an opioid epidemic dynamical model with random perturbations
Semi-Automatic RECIST Labeling on CT Scans with Cascaded Convolutional Neural Networks
A Hierarchical Deep Learning Natural Language Parser for Fashion
The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems
Testability of the exclusion restriction in continuous instrumental variable models
Optimal control of differential-algebraic equations from an ordinary differential equation perspective
Self-supervised Learning for Dense Depth Estimation in Monocular Endoscopy
SkinNet: A Deep Learning Framework for Skin Lesion Segmentation
Pushing the boundaries of parallel Deep Learning — A practical approach
Spiked covariances and principal components analysis in high-dimensional random effects models
Number of valid decompositions of Fibonacci prefixes
Parameterized algorithms and data reduction for safe convoy routing
Mapping Unparalleled Clinical Professional and Consumer Languages with Embedding Alignment
Towards Optimal Estimation of Bivariate Isotonic Matrices with Unknown Permutations
Function space bases in the dune-functions module
Learning dynamical systems with particle stochastic approximation EM
Maximum Rooted Connected Expansion
IR2VI: Enhanced Night Environmental Perception by Unsupervised Thermal Image Translation
First passage percolation in sparse random graphs with boundary weights
Asymptotic Properties of Recursive Maximum Likelihood Estimation in Non-Linear State-Space Models
Learning Single-Image Depth from Videos using Quality Assessment Networks
Traffic Differentiation in Dense WLANs with CSMA/ECA-DR MAC Protocol
On the Hausdorff dimension of a 2-dimensional Weierstrass curve
Fundamental limits of detection in the spiked Wigner model
Analyticity of Entropy Rates of Continuous-State Hidden Markov Models
Bias of Particle Approximations to Optimal Filter Derivative
Tracking Emerges by Colorizing Videos
Stability of Optimal Filter Higher-Order Derivatives
Stochastic natural gradient descent draws posterior samples in function space
Computation and Bounding of Folkman Numbers
A Machine-learning framework for automatic reference-free quality assessment in MRI