Columnar Database Techniques for Creating AI Features

Recent advances with in-memory columnar database techniques have increased the performance of analytical queries on very large databases and data warehouses. At the same time, advances in artificial intelligence (AI) algorithms have increased the ability to analyze data. We use the term AI to encompass both Deep Learning (DL or neural network) and Machine Learning (ML aka Big Data analytics). Our exploration of the AI full stack has led us to a cross-stack columnar database innovation that efficiently creates features for AI analytics. The innovation is to create Augmented Dictionary Values (ADVs) to add to existing columnar database dictionaries in order to increase the efficiency of featurization by minimizing data movement and data duplication. We show how various forms of featurization (feature selection, feature extraction, and feature creation) can be efficiently calculated in a columnar database. The full stack AI investigation has also led us to propose an integrated columnar database and AI architecture. This architecture has information flows and feedback loops to improve the whole analytics cycle during multiple iterations of extracting data from the data sources, featurization, and analysis.

Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

Learning-based pattern classifiers, including deep networks, have demonstrated impressive performance in several application domains, ranging from computer vision to computer security. However, it has also been shown that adversarial input perturbations carefully crafted either at training or at test time can easily subvert their predictions. The vulnerability of machine learning to adversarial inputs (also known as adversarial examples), along with the design of suitable countermeasures, have been investigated in the research field of adversarial machine learning. In this work, we provide a thorough overview of the evolution of this interdisciplinary research area over the last ten years, starting from pioneering, earlier work up to more recent work aimed at understanding the security properties of deep learning algorithms, in the context of different applications. We report interesting connections between these apparently-different lines of work, highlighting common misconceptions related to the evaluation of the security of machine-learning algorithms. We finally discuss the main limitations of current work, along with the corresponding future research challenges towards the design of more secure learning algorithms.

Opinion Dynamics and Social Power Evolution: A Single-Timescale Model

This paper studies the evolution of self-appraisal and social power, for a group of individuals who discuss and form opinions. We consider a modification of the recently proposed DeGroot-Friedkin (DF) model, in which the opinion formation process takes place on the same timescale as the reflected appraisal process; we call this new model the single-timescale DF model. We provide a comprehensive analysis of the equilibria and convergence properties of the model for the settings of irreducible and reducible influence networks. For the setting of irreducible influence networks, the single-timescale DF model has the same behavior as the original DF model, that is, it predicts among other things that the social power ranking among individuals is asymptotically equal to their centrality ranking, that social power tends to accumulate at the top of the centrality ranking hierarchy, and that an autocratic (resp., democratic) power structure arises when the centrality scores are maximally nonuniform (resp., uniform). For the setting of reducible influence networks, the single-timescale DF model behaves differently from the original DF model in two ways. First, an individual, who corresponds to a reducible node in a reducible influence network, can keep all social power in the single-timescale DF model if the initial condition does so, whereas its social power asymptotically vanishes in the original DF model. Second, when the associated network has multiple sinks, the two models behave very differently: the original DF model has a single globally-attractive equilibrium, whereas any partition of social power among the sinks is allowable at equilibrium in the single-timescale DF model.

Confidence Intervals for Quantiles from Histograms and Other Grouped Data

Interval estimation of quantiles has been treated by many in the literature. However, to the best of our knowledge there has been no consideration for interval estimation when the data are available in grouped format. Motivated by this, we introduce several methods to obtain confidence intervals for quantiles when only grouped data is available. Our preferred method for interval estimation is to approximate the underlying density using the Generalized Lambda Distribution (GLD) to both estimate the quantiles and variance of the quantile estimators. We compare the GLD method with some other methods that we also introduce which are based on a frequency approximation approach and a linear interpolation approximation of the density. Our methods are strongly supported by simulations showing that excellent coverage can be achieved for a wide number of distributions. These distributions include highly-skewed distributions such as the log-normal, Dagum and Singh-Maddala distributions. We also apply our methods to real data and show that inference can be carried out on published outcomes that have been summarized only by a histogram. Our methods are therefore useful for a broad range of applications. We have also created a web application that can be used to conveniently calculate the estimators.

The Approximate Duality Gap Technique: A Unified Theory of First-Order Methods

We present a general technique for the analysis of first-order methods. The technique relies on the construction of a duality gap for an appropriate approximation of the objective function, where the function approximation improves as the algorithm converges. We show that in continuous time enforcement of an invariant that this approximate duality gap decreases at a certain rate exactly recovers a wide range of first-order continuous-time methods. We characterize the discretization errors incurred by different discretization methods, and show how iteration-complexity-optimal methods for various classes of problems cancel out the discretization error. The technique is illustrated on various classes of problems — including solving variational inequalities for smooth monotone operators, convex minimization for Lipschitz-continuous objectives, smooth convex minimization, composite minimization, and smooth and strongly convex minimization — and naturally extends to other settings.

CNNs are Globally Optimal Given Multi-Layer Support

Stochastic Gradient Descent (SGD) is the central workhorse for training modern CNNs. Although giving impressive empirical performance it can be slow to converge. In this paper we explore a novel strategy for training a CNN using an alternation strategy that offers substantial speedups during training. We make the following contributions: (i) replace the ReLU non-linearity within a CNN with positive hard-thresholding, (ii) reinterpret this non-linearity as a binary state vector making the entire CNN linear if the multi-layer support is known, and (iii) demonstrate that under certain conditions a global optima to the CNN can be found through local descent. We then employ a novel alternation strategy (between weights and support) for CNN training that leads to substantially faster convergence rates, nice theoretical properties, and achieving state of the art results across large scale datasets (e.g. ImageNet) as well as other standard benchmarks.

Broadcasting Convolutional Network

While convolutional neural networks (CNNs) are widely used for handling spatio-temporal scenes, there exist limitations in reasoning relations among spatial features caused by their inherent structures, which have been issued consistently in many studies. In this paper, we propose Broadcasting Convolutional Networks (BCN) that allow global receptive fields to share spatial information. BCNs are simple network modules that collect effective spatial features, embed location informations and broadcast them to the entire feature maps without much additional computational cost. This method gains great improvements in feature localization problems through efficiently extending the receptive fields, and can easily be implemented within any structure of CNNs. We further utilize BCN to propose Multi-Relational Networks (multiRN) that greatly improve existing Relation Networks (RNs). In pixel-based relation reasoning problems, multiRN with BCNs implanted extends the concept of `pairwise relations’ from conventional RNs to `multiple relations’ by relating each object with multiple objects at once and not in pairs. This yields in O(n) complexity for n number of objects, which is a vast computational gain from RNs that take O(n^2). Through experiments, BCNs are proven for their usability on relation reasoning problems, which is due from their efficient handlings of spatial information.

Bootstrap of residual processes in regression: to smooth or not to smooth ?

In this paper we consider a location model of the form Y = m(X) + \varepsilon, where m(\cdot) is the unknown regression function, the error \varepsilon is independent of the p-dimensional covariate X and E(\varepsilon)=0. Given i.i.d. data (X_1,Y_1),\ldots,(X_n,Y_n) and given an estimator \hat m(\cdot) of the function m(\cdot) (which can be parametric or nonparametric of nature), we estimate the distribution of the error term \varepsilon by the empirical distribution of the residuals Y_i-\hat m(X_i), i=1,\ldots,n. To approximate the distribution of this estimator, Koul and Lahiri (1994) and Neumeyer (2008, 2009) proposed bootstrap procedures, based on smoothing the residuals either before or after drawing bootstrap samples. So far it has been an open question whether a classical non-smooth residual bootstrap is asymptotically valid in this context. In this paper we solve this open problem, and show that the non-smooth residual bootstrap is consistent. We illustrate this theoretical result by means of simulations, that show the accuracy of this bootstrap procedure for various models, testing procedures and sample sizes.

Group Activity Selection on Social Networks

We propose a new variant of the group activity selection problem (GASP), where the agents are placed on a social network and activities can only be assigned to connected subgroups (gGASP). We show that if multiple groups can simultaneously engage in the same activity, finding a stable outcome is easy as long as the network is acyclic. In contrast, if each activity can be assigned to a single group only, finding stable outcomes becomes computationally intractable, even if the underlying network is very simple: the problem of determining whether a given instance of a gGASP admits a Nash stable outcome turns out to be NP-hard when the social network is a path or a star, or if the size of each connected component is bounded by a constant. We then study the parameterized complexity of finding outcomes of gGASP that are Nash stable, individually stable or core stable. For the parameter `number of activities’, we propose an FPT algorithm for Nash stability for the case where the social network is acyclic and obtain a W[1]-hardness result for cliques (i.e., for standard GASP); similar results hold for individual stability. In contrast, finding a core stable outcome is hard even if the number of activities is bounded by a small constant, both for standard GASP and when the social network is a star. For the parameter `number of players’, all problems we consider are in XP for arbitrary social networks; on the other hand, we prove W[1]-hardness results with respect to the parameter `number of players’ for the case where the social network is a clique.

Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing

Deep convolutional neural network (DCNN) based supervised learning is a widely practiced approach for large-scale image classification. However, retraining these large networks to accommodate new, previously unseen data demands high computational time and energy requirements. Also, previously seen training samples may not be available at the time of retraining. We propose an efficient training methodology and incrementally growing a DCNN to allow new classes to be learned while sharing part of the base network. Our proposed methodology is inspired by transfer learning techniques, although it does not forget previously learned classes. An updated network for learning new set of classes is formed using previously learned convolutional layers (shared from initial part of base network) with addition of few newly added convolutional kernels included in the later layers of the network. We evaluated the proposed scheme on several recognition applications. The classification accuracy achieved by our approach is comparable to the regular incremental learning approach (where networks are updated with new training samples only, without any network sharing).

End-to-end Learning of Deterministic Decision Trees

Conventional decision trees have a number of favorable properties, including interpretability, a small computational footprint and the ability to learn from little training data. However, they lack a key quality that has helped fuel the deep learning revolution: that of being end-to-end trainable, and to learn from scratch those features that best allow to solve a given supervised learning problem. Recent work (Kontschieder 2015) has addressed this deficit, but at the cost of losing a main attractive trait of decision trees: the fact that each sample is routed along a small subset of tree nodes only. We here propose a model and Expectation-Maximization training scheme for decision trees that are fully probabilistic at train time, but after a deterministic annealing process become deterministic at test time. We also analyze the learned oblique split parameters on image datasets and show that Neural Networks can be trained at each split node. In summary, we present the first end-to-end learning scheme for deterministic decision trees and present results on par with or superior to published standard oblique decision tree algorithms.

A Heuristic Search Algorithm Using the Stability of Learning Algorithms as the Fitness Function in Certain Scenarios: An Artificial General Intelligence Engineering Approach

This paper presents a non-manual design engineering method based on heuristic search algorithm to search for candidate agents in the solution space which formed by artificial intelligence agents modeled on the base of bionics.Compared with the artificial design method represented by meta-learning and the bionics method represented by the neural architecture chip,this method is more feasible for realizing artificial general intelligence,and it has a much better interaction with cognitive neuroscience;at the same time,the engineering method is based on the theoretical hypothesis that the final learning algorithm is stable in certain scenarios,and has generalization ability in various scenarios.The paper discusses the theory preliminarily and proposes the possible correlation between the theory and the fixed-point theorem in the field of mathematics.Limited by the author’s knowledge level,this correlation is proposed only as a kind of conjecture.

Effective Extensible Programming: Unleashing Julia on GPUs

GPUs and other accelerators are popular devices for accelerating compute-intensive, parallelizable applications. However, programming these devices is a difficult task. Writing efficient device code is challenging, and is typically done in a low-level programming language. High-level languages are rarely supported, or do not integrate with the rest of the high-level language ecosystem. To overcome this, we propose compiler infrastructure to efficiently add support for new hardware or environments to an existing programming language. We evaluate our approach by adding support for NVIDIA GPUs to the Julia programming language. By integrating with the existing compiler, we significantly lower the cost to implement and maintain the new compiler, and facilitate reuse of existing application code. Moreover, use of the high-level Julia programming language enables new and dynamic approaches for GPU programming. This greatly improves programmer productivity, while maintaining application performance similar to that of the official NVIDIA CUDA toolkit.

Achieving the time of $1$-NN, but the accuracy of $k$-NN
A note on searching sorted unbalanced three-dimensional arrays
A Note on Cohamiltonian Graphs
On overfitting and post-selection uncertainty assessments
Predictive inference for locally stationary time series with an application to climate data
Connected power domination in graphs
Noisy Natural Gradient as Variational Inference
Conflict-driven Hybrid Observer-based Anomaly Detection
Top-down Flow Transformer Networks
The oriented size Ramsey number of directed paths
Deep Regionlets for Object Detection
Estimating the error variance in a high-dimensional linear model
Pieri rules for the Jack polynomials in superspace and the 6-vertex model
Tomographic Reconstruction using Global Statistical Prior
Compressive Phase Retrieval via Reweighted Amplitude Flow
High performance ultra-low-precision convolutions on mobile devices
Emergence of a spectral gap in a class of random matrices associated with split graphs
Sparse learning of stochastic dynamic equations
A Novel Model for Arbitration between Planning and Habitual Control Systems
Targeted Random Projection for Prediction from High-Dimensional Features
HyperPower: Power- and Memory-Constrained Hyper-Parameter Optimization for Neural Networks
On Colouring $(2P_2,H)$-Free and $(P_5,H)$-Free Graphs
Quantifying how much sensory information in a neural code is relevant for behavior
Covers and partial transversals of Latin squares
CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognition
On Sub-Packetization of Capacity-Achieving PIR Schemes for MDS Coded Databases
Deep Primal-Dual Reinforcement Learning: Accelerating Actor-Critic using Bellman Duality
Asymptotic coverage probabilities of bootstrap percentile confidence intervals for constrained parameters
Gathering in the Plane of Location-Aware Robots in the Presence of Spies
Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal
A Corpus of Deep Argumentative Structures as an Explanation to Argumentative Relations
Cost-sensitive detection with variational autoencoders for environmental acoustic sensing
Sub-measures and applications to wedge intersection of positive closed currents and complex dynamics
Adversarial Examples that Fool Detectors
Linear $d$-polychromatic $Q_{d-1}$-colorings of the Hypercube
Multiplicative Coevolution Regression Models for Longitudinal Networks and Nodal Attributes
Take it in your stride: Do we need striding in CNNs?
Cooperative Control via Congestion Game Approach
Semi-Supervised Learning with IPM-based GANs: an Empirical Study
The Representation Theory of 2-Sylow Subgroups of the Symmetric Group
Gini-regularized Optimal Transport with an Application to Spatio-Temporal Forecasting
TV-GAN: Generative Adversarial Network Based Thermal to Visible Face Recognition
Canonizing structural Ramsey theorems
Convergence Rates of Variational Posterior Distributions
Progress on the Frobenius Coin Problem. A Classical Approach and a Record-Theoretic Approach
High Dimensional Expanders
Learning Random Fourier Features by Hybrid Constrained Optimization
Parallel Prefix Algorithms for the Registration of Arbitrarily Long Electron Micrograph Series
Fair mixing: the case of dichotomous preferences
Cutpoints for Random Walks on Quasi-Transitive Graphs
Distributed learning of CNNs on heterogeneous CPU/GPU architectures
Hungarian Layer: Logics Empowered Neural Architecture
Maximum Classifier Discrepancy for Unsupervised Domain Adaptation
Multi-layer Random Walks Synchronization for Multi-attributed Multiple Graph Matching
Index Reduction for Differential-Algebraic Equations with Mixed Matrices
Gaussian Process Regression for In-situ Capacity Estimation of Lithium-ion Batteries
Blind Multi-user Detection for Autonomous Grant-free High-Overloading MA without Reference Signal
New Methods for Small Area Estimation with Linkage Uncertainty
Solving internal covariate shift in deep learning with linked neurons
Sets of Stochastic Matrices with Converging Products: Bounds and Complexity
In-Place Activated BatchNorm for Memory-Optimized Training of DNNs
Disentangled Person Image Generation
The multiproximal linearization method for convex composite problems
Many-body localization of bosons in optical lattices
High-dimensional robust regression and outliers detection with SLOPE
On permutation characters and Sylow $p$-subgroups of $\mathfrak{S}_n$
Using SVDD in SimpleMKL for 3D-Shapes Filtering
Creating Capsule Wardrobes from Fashion Images
A combinatorial formula for graded multiplicities in excellent filtrations
Is My Model Flexible Enough? Information-Theoretic Model Check
Orientable $\mathbb{Z}{}_{n}$-distance magic regular graphs
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
Feedback Capacity and Coding for the $(0,k)$-RLL Input-Constrained BEC
milliProxy: a TCP Proxy Architecture for 5G mmWave Cellular Systems
A duality principle for non-linear elasticity
On the metric dimension of Cartesian powers of a graph
Opportunities for Mining Radiology Archives for Pediatric Control Images
ChemNet: A Transferable and Generalizable Deep Neural Network for Small-Molecule Property Prediction
Density bounds for solutions to differential equations driven by Gaussian rough paths
Model and Discretization Error Adaptivity within Stationary Gas Transport Optimization
Dimension-free PAC-Bayesian bounds for matrices, vectors, and linear least squares regression
Exact active subspace Metropolis-Hastings, with applications to the Lorenz-96 system
A Convergence Diagnostic for Bayesian Clustering
On the Duality Between Retinex and Image Dehazing
Constant curvature metrics for Markov chains
Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs
A wavelet Plancherel theory with application to sparse continuous wavelet transform
A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations
On Usage of Autoencoders and Siamese Networks for Online Handwritten Signature Verification
A Deep Network Model for Paraphrase Detection in Short Text Messages
Hybrid eye center localization using cascaded regression and hand-crafted model fitting
Stacked Denoising Autoencoders and Transfer Learning for Immunogold Particles Detection and Recognition
Core Discovery in Hidden Graphs
On the second largest component of random hyperbolic graphs
RelNN: A Deep Neural Model for Relational Learning
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient
The Shrinkage Variance Hotelling $T^2$ Test for Genomic Profiling Studies
Stability of Control Systems with Feedback from Energy Harvesting Sensors
Strong convergence of quantum random walks via semigroup decomposition
Sketched Clustering via Hybrid Approximate Message Passing
t-Private Information Retrieval Schemes Using Transitive Codes
Stochastic reconstruction of an oolitic limestone by generative adversarial networks
Effective Neural Solution for Multi-Criteria Word Segmentation
Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz
Remarks on Bayesian Control Charts
Per-Pixel Feedback for improving Semantic Segmentation
MoDL: Model Based Deep Learning Architecture for Inverse Problems
Learned Perceptual Image Enhancement
Perspectival Knowledge in PSOA RuleML: Representation, Model Theory, and Translation
Approximate public-signal correlated equilibria for nonzero-sum differential games
Multi-Scale Video Frame-Synthesis Network with Transitive Consistency Loss
Optimal Scale Invariant Wigner Spectrum Estimation of Gaussian Locally Self-Similar Processes Using Hermite Functions
An End to End Deep Neural Network for Iris Segmentation in Unconstraint Scenarios
Median inverse problem and approximating the number of $k$-median inverses of a permutation
Network Analysis for Explanation
Circuit Preserving Edge Maps II
Deep Image Smoothing based on Texture and Structure Guidance
Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies
Representations of Sound in Deep Learning of Audio Features from Music
A Taxonomy and Future Directions for Sustainable Cloud Computing: 360 Degree View
Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start
Blind Multi-class Ensemble Learning with Unequally Reliable Classifiers
Linked systems of symmetric designs
Exploiting Modern Hardware for High-Dimensional Nearest Neighbor Search
On Topology Optimization and Canonical Duality Method
Chaining Identity Mapping Modules for Image Denoising
Layered Non-Orthogonal Random Access with SIC and Transmit Diversity for Reliable Transmissions
Dense Optical Flow based Change Detection Network Robust to Difference of Camera Viewpoints
OneDataShare: A Vision for Cloud-hosted Data Transfer Scheduling and Optimization as a Service
CycleGAN: a Master of Steganography
Compact Hash Code Learning with Binary Deep Neural Network
Sequence to Sequence Networks for Roman-Urdu to Urdu Transliteration
Shape from Shading through Shape Evolution
On Skew-Symmetric Games
Bayesian Variable Selection in High Dimensional Survival Time Cancer Genomic Datasets using Nonlocal Priors
Deciding the existence of a cherry-picking sequence is hard on two trees
Learning 2D Gabor Filters by Infinite Kernel Learning Regression
Recruitment Market Trend Analysis with Sequential Latent Variable Models
Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser
Periortree: An Extention of R-Tree for Periodic Boundary Conditions
Nonconvex Sparse Spectral Clustering by Alternating Direction Method of Multipliers and Its Convergence Analysis
Direct and Real-Time Cardiovascular Risk Prediction
A Classification of Functions in Multiterminal Distributed Computing
Collaborative Company Profiling: Insights from an Employee’s Perspective
Censored pairwise likelihood-based tests for mixing coefficient of spatial max-mixture models
Compression of Wannier functions into Gaussian-type orbitals
Reconstruction of Brain Activity from EEG/MEG Using MV-PURE Framework
Recurrence of Multidimensional Persistent Random Walks. Fourier and Series Criteria
A Descent Method for Equality and Inequality Constrained Multiobjective Optimization Problems
Stochastic Dual Coordinate Descent with Bandit Sampling
Graph Theory Based Approach to Users Grouping and Downlink Scheduling in FDD Massive MIMO
The critical greedy server on the integers is recurrent
p-Values for Credibility
The Graph Curvature Calculator and the curvatures of cubic graphs
A Frequency Domain Neural Network for Fast Image Super-resolution
Shrewd Selection Speeds Surfing: Use Smart EXP3!
Approximation intensity for pairwise interaction Gibbs point processes using determinantal point processes
Artificial Neural Networks that Learn to Satisfy Logic Constraints
Corotational Cut Finite Element Method for real-time surgical simulation: application to needle insertion simulation
Iterated filtering methods for Markov process epidemic models
Energy Efficient and Throughput Optimal CSMA Scheme
A separation between tropical matrix ranks
An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing
Image Inpainting for High-Resolution Textures using CNN Texture Synthesis
Weighted Committee Games
Tilings of the plane with unit area triangles of bounded diameter
Detecting confounding due to subject identification in clinical machine learning diagnostic applications: a permutation test approach
Simultaneous Hand Pose and Skeleton Bone-Lengths Estimation from a Single Depth Image
Asymptotic expansion for vector-valued sequences of random variables with focus on Wiener chaos
Symmetric, Hankel-symmetric, and Centrosymmetric Doubly Stochastic Matrices
A Class of Logistic Functions for Approximating State-Inclusive Koopman Operators
Building competitive direct acoustics-to-word models for English conversational speech recognition
On Adaptive Estimation for Dynamic Bernoulli Bandits
Monochromatic cycle covers in random graphs
Weaving Multi-scale Context for Single Shot Detector
On the complexity of graph coloring with additional local conditions
Combining Deep Universal Features, Semantic Attributes, and Hierarchical Classification for Zero-Shot Learning
Structure-Adaptive, Variance-Reduced, and Accelerated Stochastic Optimization
Graph-based time-space trade-offs for approximate near neighbors
Minimal Solvers for Monocular Rolling Shutter Compensation under Ackermann Motion
Qualitative Stability and Synchronicity Analysis of Power Network Models in Port-Hamiltonian form
Class Rectification Hard Mining for Imbalanced Deep Learning
A Method for Finding Similar Documents Relying on Adding Repetition of Symbols in Length Based Filtering
On inverse powers of graphs and topological implications of Hedetniemi’s conjecture
Using simulation studies to evaluate statistical methods
Characterizing the hyper-parameter space of LSTM language models for mixed context applications
On the connection between a skew product IFS and the ergodic optimization for a finite family of potentials
Convergence and non-negativity preserving of the solution of balanced method for the delay CIR model with jump
Uniquely $K^{(k)}_r$-saturated Hypergraphs
Basic Thresholding Classification