In this article, we’ll go over the math behind the categorical distribution, the algebraic structure of the distribution, and how to manipulate it within Haskell’s HLearn library. We’ll also see some examples of how this focus on algebra makes HLearn’s interface more powerful than other common statistical packages. Everything that we’re going to see is in a certain sense very “obvious” to a statistician, but this algebraic framework also makes it convenient. And since programmers are inherently lazy, this is a Very Good Thing.

I have analysed a dataset of 974 LinkedIn job advertisements for data scientists, based in the US. The skills listed in the dataset are classified as ‘cloud_software_required’,’database_software_required’, ‘statistic_software_required’, and ‘programming_language_required’.

Tools and approaches to response modeling are so fully explored and understood there’s nothing new left to be learned, right? Get the best lift curve with the best Gini coefficient and run with it. Actually, for a long time we’ve had that niggling little bit of doubt about who would have bought anyway even without our special offer. Or worse, in churn prevention, did our campaign actually awaken a discontented customer and cause them to reevaluate their relationship with us. Both of these concerns are no doubt true. For some customers no promotion was necessary to get them to buy, and yes we undoubtedly poked some sleeping bears who rose from their slumber and used this reminder to run off with our competition. There is a technique that corrects for this and offers much improved campaign ROIs. It’s called Uplift Modeling.

At Dataquest, our mission is to help prepare people for data science roles in companies. This means that we mainly teach Python. However, due to its visualization and statistical libraries, R can augment your data science workflow and help you explore data more effectively.

Sophisticated data visualizations are pushing the bounds of what we can process, sometimes to the breaking point. What are the signature styles of contemporary data vis, and will they stand the test of time?

Network analysis offers a new set of techniques to tackle the persistent and growing problem of complex fraud. Network analysis supplements traditional techniques by providing a mechanism to bridge investigative and analytics methods. Beyond base visualization, network analysis provides a standardized platform for complex fraud pattern storage and retrieval, pattern discovery and detection, statistical analysis, and risk scoring. This article gives an overview of the main challenges and demonstrates a promising approach using a hands-on example.

The Caterpillar Tube Pricing competition asked teams to use detailed tube, component, and volume data to predict the price a supplier would quote for the manufacturing of different tube assemblies. Team ‘Gilberto | Josef | Leustago | Mario’ finished in first place, bringing in new players (with new models) near the team merger deadline to create a strong ensemble. Feature engineering played a key role in developing their individual models, and team discussions in the last week of the competition brought them to the top of the leaderboard.

The essential mathematics necessary for Data Science can be acquired with these 15 MOOCs, with a strong emphasis on applied algebra & statistics.

Here, dplyr uses non-standard evaluation in finding the contents for mpg and wt, knowing that it needs to look in the context of mtcars. This is nice for interactive use, but not so nice for using mutate inside a function where mpg and wt are inputs to the function. The goal is to write a function f that takes the columns in mtcars you want to add up as strings, and executes mutate. Note that we also want to be able to set the new column name. A first naive approach might be:

Neural networks have always been one of the most fascinating machine learning model in my opinion, not only because of the fancy backpropagation algorithm, but also because of their complexity (think of deep learning with many hidden layers) and structure inspired by the brain. Neural networks have not always been popular, partly because they were, and still are in some cases, computationally expensive and partly because they did not seem to yield better results when compared with simpler methods such as support vector machines (SVMs). Nevertheless Neural Newtorks have, once again, raised attention and become popular.

This post is actually a continuation of the previous post, and is motivated by this article that discusses the graphics and statistical analysis for a two treatment, two period, two sequence (2x2x2) crossover drug interaction study of a new treatment versus the standard. Whereas the previous post was devoted to implementing some of the graphics presented in the article, in this post we try to recreate the statistical analysis calculations for the data from the drug interaction study. The statistical analysis is implemented with R.

There are four key data visualization techniques used by data analysis pros in the government and local law enforcement. As financial institutions, e-commerce organizations and social network analysts begin to apply data visualization more frequently, these techniques will help guide the process of uncovering meaningful insights hidden within mountains of disparate data. This post focuses on advanced data visualization using relationship graphs.