A Big Data Analyzer for Large Trace Logs

Current generation of Internet-based services are typically hosted on large data centers that take the form of warehouse-size structures housing tens of thousands of servers. Continued availability of a modern data center is the result of a complex orchestration among many internal and external actors including computing hardware, multiple layers of intricate software, networking and storage devices, electrical power and cooling plants. During the course of their operation, many of these components produce large amounts of data in the form of event and error logs that are essential not only for identifying and resolving problems but also for improving data center efficiency and management. Most of these activities would benefit significantly from data analytics techniques to exploit hidden statistical patterns and correlations that may be present in the data. The sheer volume of data to be analyzed makes uncovering these correlations and patterns a challenging task. This paper presents BiDAl, a prototype Java tool for log-data analysis that incorporates several Big Data technologies in order to simplify the task of extracting information from data traces produced by large clusters and server farms. BiDAl provides the user with several analysis languages (SQL, R and Hadoop MapReduce) and storage backends (HDFS and SQLite) that can be freely mixed and matched so that a custom tool for a specific task can be easily constructed. BiDAl has a modular architecture so that it can be extended with other backends and analysis languages in the future. In this paper we present the design of BiDAl and describe our experience using it to analyze publicly-available traces from Google data clusters, with the goal of building a realistic model of a complex data center.

A DEEP analysis of the META-DES framework for dynamic selection of ensemble of classifiers

Dynamic ensemble selection (DES) techniques work by estimating the level of competence of each classifier from a pool of classifiers. Only the most competent ones are selected to classify a given test sample. Hence, the key issue in DES is the criterion used to estimate the level of competence of the classifiers to predict the label of a given test sample. In order to perform a more robust ensemble selection, we proposed the META-DES framework using meta-learning, where multiple criteria are encoded as meta-features and are passed down to a meta-classifier that is trained to estimate the competence level of a given classifier. In this technical report, we present a step-by-step analysis of each phase of the framework during training and test. We show how the sets meta-features are extracted as well as the impact of each set of meta-features in the estimation of the competence level of the base classifier. Moreover, an analysis of the impact of several factors in the system performance, such as, the number of classifiers in the pool, the use of different linear base classifiers as well as the size of the validation data. We show that using dynamic selection of linear classifiers through the META-DES framework we can solve complex non-linear classification problems where other combination techniques such as AdaBoost cannot.

A Neural Attention Model for Abstractive Sentence Summarization

Summarization based on text extraction is inherently limited, but generation-style abstractive methods have proven challenging to build. In this work, we propose a fully data-driven approach to abstractive sentence summarization. Our method utilizes a local attention-based model that generates each word of the summary conditioned on the input sentence. While the model is structurally simple, it can easily be trained end-to-end and scales to a large amount of training data. The model shows significant performance gains on the DUC-2004 shared task compared with several strong baselines.

A sustainability condition for stochastic forest model

A stochastic forest model for young and old age trees is studied. First, we prove existence, uniqueness and boundedness of global nonnegative solutions. Second, we investigate asymptotic behavior of global solutions by giving a sufficient condition for sustainability of forest. Under this condition, we also show existence of a Borel invariant measure. Third, we present sufficient conditions for decline of forest. Finally, we give some numerical examples.

Analysis of Communication Pattern with Scammers in Enron Corpus

This paper is an exploratory analysis into fraud detection taking Enron email corpus as the case study. The paper posits conclusions like strict servitude and unquestionable faith among employees as breeding grounds for sham among higher executives. We also try to infer on the nature of communication between fraudulent employees and between non- fraudulent-fraudulent employees

Bayesian Change Point Analysis of Linear Models on Graphs

Consider observations y_1,\dots,y_n on nodes of a connected graph, where the y_i independently come from N(\theta_i, \sigma^2) distributions and an unknown partition divides the n observations into blocks. One well-studied class of change point problems assumes the means \theta_i are equal for all nodes within contiguous blocks of a simple graph of sequential observations; both frequentist and Bayesian approaches have been used to estimate the \theta_i and the change points of the underlying partition. This paper examines a broad class of change point problems on general connected graphs in which a regression model is assumed to apply within each block of the partition of the graph. This general class also supports multivariate change point problems. We use Bayesian methods to estimate change points or block boundaries of the underlying partition. This paper presents the methodology for the general class of change point problems and develops new algorithms for implementation via Markov Chain Monte Carlo. The paper concludes with simulations and real data examples to demonstrate application of the methodology on a wide range of problems.

Evaluating user reputation in online rating systems via an iterative group-based ranking method

Reputation is a valuable asset in online social lives and it has drawn increased attention. How to evaluate user reputation in online rating systems is especially significant due to the existence of spamming attacks. To address this issue, so far, a variety of methods have been proposed, including network-based methods, quality-based methods and group-based ranking method. In this paper, we propose an iterative group-based ranking (IGR) method by introducing an iterative reputation-allocation process into the original group-based ranking (GR) method. More specifically, users with higher reputation have higher weights in dominating the corresponding group sizes. The reputation of users and the corresponding group sizes are iteratively updated until they become stable. Results on two real data sets suggest that the proposed IGR method has better performance and its robustness is considerably improved comparing with the original GR method. Our work highlights the positive role of users’ grouping behavior towards a better reputation evaluation.

Importance Weighted Autoencoders

The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It makes two strong assumptions about posterior inference: that the posterior distribution is approximately factorial, and that its parameters can be approximated with nonlinear regression from the observations. As we show empirically, the VAE objective can lead to overly simplified representations which fail to use the network’s entire modeling capacity. We present the importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting. In the IWAE, the recognition network uses multiple samples to approximate the posterior, giving it increased flexibility to model complex posteriors which do not fit the VAE modeling assumptions. We show empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log-likelihood on density estimation benchmarks.

Parsimonious Time Series Clustering

We introduce a parsimonious model-based framework for clustering time course data. In these applications the computational burden becomes often an issue due to the number of available observations. The measured time series can also be very noisy and sparse and a suitable model describing them can be hard to define. We propose to model the observed measurements by using P-spline smoothers and to cluster the functional objects as summarized by the optimal spline coefficients. In principle, this idea can be adopted within all the most common clustering frameworks. In this work we discuss applications based on a k-means algorithm. We evaluate the accuracy and the efficiency of our proposal by simulations and by dealing with drosophila melanogaster gene expression data.

Sensor-Type Classification in Buildings

Many sensors/meters are deployed in commercial buildings to monitor and optimize their performance. However, because sensor metadata is inconsistent across buildings, software-based solutions are tightly coupled to the sensor metadata conventions (i.e. schemas and naming) for each building. Running the same software across buildings requires significant integration effort. Metadata normalization is critical for scaling the deployment process and allows us to decouple building-specific conventions from the code written for building applications. It also allows us to deal with missing metadata. One important aspect of normalization is to differentiate sensors by the typeof phenomena being observed. In this paper, we propose a general, simple, yet effective classification scheme to differentiate sensors in buildings by type. We perform ensemble learning on data collected from over 2000 sensor streams in two buildings. Our approach is able to achieve more than 92% accuracy for classification within buildings and more than 82% accuracy for across buildings. We also introduce a method for identifying potential misclassified streams. This is important because it allows us to identify opportunities to attain more input from experts — input that could help improve classification accuracy when ground truth is unavailable. We show that by adjusting a threshold value we are able to identify at least 30% of the misclassified instances.

Statistical Inference for Partially Observed Markov Processes via the R Package pomp

Partially observed Markov process (POMP) models, also known as hidden Markov models or state space models, are ubiquitous tools for time series analysis. The R package pomp provides a very flexible framework for Monte Carlo statistical investigations using nonlinear, non-Gaussian POMP models. A range of modern statistical methods for POMP models have been implemented in this framework including sequential Monte Carlo, iterated filtering, particle Markov chain Monte Carlo, approximate Bayesian computation, maximum synthetic likelihood estimation, nonlinear forecasting, and trajectory matching. In this paper, we demonstrate the application of these methodologies using some simple toy problems. We also illustrate the specification of more complex POMP models, using a nonlinear epidemiological model with a discrete population, seasonality, and extra-demographic stochasticity. We discuss the specification of user-defined models and the development of additional methods within the programming environment provided by pomp.

A classical limit of Noumi’s q-integral operator

A Condition for Distinguishing Sceneries on Non-abelian Groups

A family of $m$-ovoids of parabolic quadrics

A hybrid COA-DEA method for solving multi-objective problems

A note on Probably Certifiably Correct algorithms

About spectrum of $T_2$-quasigroups

Adjusted Concordance Index, an extension of the Adjusted Rand index to fuzzy partitions

Bayesian sequential testing of the drift of a Brownian motion

Characterisation of symmetries of unlabelled triangulations and its applications

Chromatic Ramsey number of acyclic hypergraphs

Clustering of high values in random fields

Constructing a Family of 4-Critical Planar Graphs with High Edge-Density

Correlated Poisson processes and self-decomposable laws

Coupling Identical 1D Many-Body Localized Systems

Edge- and vertex-reinforced random walks with super-linear reinforcement on infinite graphs

Enhancement and Recognition of Reverberant and Noisy Speech by Extending Its Coherence

Entrainment and Synchronization in Heterogeneous Networks of Haken-Kelso-Bunz (HKB) Oscillators

Estimation of delta-contaminated density of the random intensity of Poisson data

Extremal Distances for Subtree Transfer Operations in Binary Trees

Finding Near-Optimal Independent Sets at Scale

Gaussian random projections for Euclidean membership problems

Heavy-tailed Independent Component Analysis

Improving the accuracy of likelihood-based inference in meta-analysis and meta-regression

L-Drawings of Directed Graphs

Mean-square convergence of the BDF2-Maruyama and backward Euler schemes for SDE satisfying a global monotonicity condition

Nonlocal effects and counter measures in cascading failures

Obtaining highly-excited eigenstates of many-body localized Hamiltonians by the density matrix renormalization group

On Record Cayley Graphs of Diameter Two

On the Gap and Time Interval between the First Two Maxima of Long Continuous Time Random Walks

On Transitive Consistency for Linear Invertible Transformations between Euclidean Coordinate Systems

Optimal Bandwidth Selection for the Fuzzy Regression Discontinuity Estimator

Order-from-disorder and critical scalings of spontaneous magnetization in random-field quantum spin systems

Recursive Markov Process

Sandpiles on the heptagonal tiling

Scheduling of Sport Tournaments with Absences

Spatial Point Pattern Analysis of the Unidentified Aerial Phenomena in France

Supports of simple modules in cyclotomic Cherednik categories O

The generic minimal rigidity of a partially triangulated torus

The maximum number of perfect matchings of semi-regular graphs

The Top-K Tau-Path Screen for Monotone Association

TMB: Automatic Differentiation and Laplace Approximation

Toward an effort estimation model for software projects integrating risk

Turing’s Imitation Game has been Improved

Variants of Plane Diameter Completion

What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment

X-Ray Photon Correlation Spectroscopy Reveals Intermittent Aging Dynamics in a Metallic Glass