Introduction

Wow, growth hacking and upselling all in the same article? Also Python.

Okay, let's start at the beginning. Imagine the following scenario: You're responsible for increasing the amount of money users spend on your e-commerce platform.

You and your team come up with different measures you could implement to achieve your goal. Two of these measures could be:

Provide a discount on your best-selling items,
Implement a rewards program that incentivices repeat purchases.

Both of these measures are fairly complex with each incurring a certain, probably known, amount of cost and an unknown effect on your customers' spending behaviour.

To decide which of these two possible measures is worth both the effort and incurred cost you need to estimate their effect on customer spend.

A natural way of estimating this effect is computing the following:

$\textrm{avg}(\textrm{spend} | \textrm{treatment} = 1) - \textrm{avg}(\textrm{spend} | \textrm{treatment} = 0) = \textrm{ATE}$.

Essentially you would compute the average spend of users who received the treatment (received a discount or signed up for rewards) and subtract from that the average spend of users who didn't receive the treatment.

Without discussing the details of the underlying potential outcomes framework, the above expression is called the average treatment effect (ATE).

Let's estimate the average treatment effect and make a decision!

So now we'll just analyze our e-commerce data of treated and untreated customers and compute the average treatment effect (ATE) for each proposed measure, right? Right?

Before you rush ahead with your ATE computations - now is a good time to take a step back and contemplate how your data was generated in the first place(data-generating process).

References and further material ...

Before we continue: My example here is based on a tutorial by the authors of the excellent DoWhy library. You can find the original tutorial here:

https://github.com/microsoft/dowhy/blob/master/docs/source/example_notebooks/dowhy_example_effect_of_memberrewards_program.ipynb

And more on DoWhy here: https://microsoft.github.io/dowhy/

Install and load libraries

!pip install dowhy --quiet

import random

import pandas as pd
import numpy as np

np.random.seed(42)
random.seed(42)

Randomized controlled trial / experimental data

So where were we ... ah right! Where does our e-commerece data come from?

Since we don't actually run an e-commerce operation here we will have to simulate our data (remember: these ideas are based on the above DoWhy tutorial).

Imagine we observe the monthly spend of each of our 10,000 users over the course of a year. Each user will spend with a certain distribution (here, a Poisson distribution) and there are both high and low spenders with different mean spends.

Over the course of the year, each user can sign up to our rewards program in any month and once they have signed up their spend goes up by 50% relative to what they would've spent without signing up.

So far so mundane: Different customers show different spending behaviour and signing up to our rewards program increases their spend.

Now the big question is: How are treatment assignment (rewards program signup) and outcome (spending behaviour) related?

If treatment and outcome, interpreted as random variables, are independent of one another then according to the potential outcome framework we can compute the ATE as easily as shown above:

$\textrm{ATE} = \textrm{avg}(\textrm{spend} | \textrm{treatment} = 1) - \textrm{avg}(\textrm{spend} | \textrm{treatment} = 0)$

When are treatment and outcome independent? The gold standard for achieving their independence in a data set is the randomized controlled trial (RCT).

In our scenario what an RCT would look like is randomly signing up our users to our rewards program - indepndent of their spending behaviour or any other characteristic.

So we would go through our list of 10,000 users and flip a coin for each of them, sign them up to our program in a random month of the year based on our coin, and send them on their merry way to continue buying stuff in our online shop.

Let's put all of this into a bit of code that simulates the spending behaviour of our users according to our thought experiment:

num_users = 10000
num_months = 12

df = pd.DataFrame({
    'user_id': np.repeat(np.arange(num_users), num_months),
    'month': np.tile(np.arange(1, num_months+1), num_users), # months are from 1 to 12
    'high_spender': np.repeat(np.random.randint(0, 2, size=num_users), num_months),
})

df['spend'] = None
df.loc[df['high_spender'] == 0, 'spend'] = np.random.poisson(250, df.loc[df['high_spender'] == 0].shape[0])
df.loc[df['high_spender'] == 1, 'spend'] = np.random.poisson(750, df.loc[df['high_spender'] == 1].shape[0])
df["spend"] = df["spend"] - df["month"] * 10

signup_months = np.random.choice(
    np.arange(1, num_months),
    num_users
) * np.random.randint(0, 2, size=num_users)  # signup_months == 0 means customer did not sign up

df['signup_month'] = np.repeat(signup_months, num_months)

# A customer is in the treatment group if and only if they signed up
df["treatment"] = df["signup_month"] > 0

# Simulating a simple treatment effect of 50%
after_signup = (df["signup_month"] < df["month"]) & (df["treatment"])
df.loc[after_signup, "spend"] = df[after_signup]["spend"] * 1.5

Let's look at user 0 and their treatment assignment as well as spend (since we're sampling random variables here you'll see something different from me):

df.loc[df['user_id'] == 0]

The effect we're interested in is the impact of rewards signup on spending behaviour - i.e. the effect on post-signup spend.

Since customers can sign up any month of the year, we'll choose one month at random and compute the effect with respect to that one month.

So let's create a new table from our time series where we collect post-signup spend for those customers that signed up in month = 6 alongside the spend of customers who never signed up.

month = 6

post_signup_spend = (
    df[df.signup_month.isin([0, month])]
    .groupby(["user_id", "signup_month", "treatment"])
    .apply(
        lambda x: pd.Series(
            {
                "post_spend": x.loc[x.month > month, "spend"].mean(),
            }
        )
    )
    .reset_index()
)
print(post_signup_spend)

      user_id  signup_month  treatment  post_spend
0           0             0      False  152.833333
1           3             0      False  162.166667
2           4             0      False  146.333333
3           6             0      False  153.666667
4           7             6       True  240.750000
...       ...           ...        ...         ...
5451     9990             0      False  629.833333
5452     9993             0      False  674.500000
5453     9994             0      False  681.000000
5454     9995             0      False  641.333333
5455     9998             0      False  658.833333

[5456 rows x 4 columns]

To get the average treatment effect (ATE) of our rewards signup treatment we now compute the average post-signup spend of the customers who signed up and subtract from that the average spend of users who didn't sign up:

post_spend = post_signup_spend\
  .groupby('treatment')\
  .agg({'post_spend': 'mean'})

post_spend

So the ATE of rewards signup on post-signup spend is:

post_spend.loc[True, 'post_spend'] - post_spend.loc[False, 'post_spend']

206.62813242372852

Since we simulated the treatment effect ourselves (50% post-signup spend increase) let's see if we can recover this effect from our data:

post_spend.loc[True, 'post_spend'] / post_spend.loc[False, 'post_spend']

1.5120740154875112

The post-signup spend for treated customers is roughly 50% greater than the spend for untreated customers - exactly the treatment effect we simulated!

Remember, however, that we are dealing with clean experimental data from a randomized controlled trial (RCT) here! The potential outcome framework tells us that for data from an RCT the simple ATE formula we used here yields the correct treatment effect due to independence of treatment assignment and outcome.

So the fact that we recovered the actual (simulated) treatment effect is nice to see but not surprising.

The issue with randomized controlled trials and observational data

Our above thought experiment where we randomly assigned our customers to our rewards program isn't very realistic.

Randomly signing up paying customers to rewards programs without their consent may upset some and may not even be permissible. The same issue with randomized treatment assignment pops up everywhere - clean randomized controlled trials are oftentimes too expensive, infeasible to implement, unethical, or not permitted.

But since we still need to experiment with our shop to drive spending behaviour we'll still go ahead and implement our rewards program. Only that this time we'll place a regular signup page in our shop where our customers can decide for themselves if they want to sign up or not.

Activating our signup page and simply observing how users and their spend behaves gives us observational data.

We usually call "observational data" just "data" without giving much thought to where they came from. I mean we've all dealt with lots of different kinds of data (marketing data, R&D measurements, HR data, etc.) and all these data were simply "observed" and didn't come out of a carefully set up experiment.

Simulating our observational data we've got the same 10,000 customers over a span of a year. We still have the same high and low spenders.

Only that now our high spenders are far more likely to sign up to our rewards program than our low spenders. My reasoning for this is that customers who spend more are also more likely to show greater brand loyalty towards us and our rewards program. Further, they visit our shop more frequently hence are more likely to notice our new rewards program and the signup page. We could also add this behaviour as random variables to our simulation below but just take a shortcut and give low spenders a 5% chance of signing up and high spenders a 95% chance.

num_users = 10000
num_months = 12

df = pd.DataFrame({
    'user_id': np.repeat(np.arange(num_users), num_months),
    'month': np.tile(np.arange(1, num_months+1), num_users), # months are from 1 to 12
    'high_spender': np.repeat(np.random.randint(0, 2, size=num_users), num_months),
})

df['spend'] = None
df.loc[df['high_spender'] == 0, 'spend'] = np.random.poisson(250, df.loc[df['high_spender'] == 0].shape[0])
df.loc[df['high_spender'] == 1, 'spend'] = np.random.poisson(750, df.loc[df['high_spender'] == 1].shape[0])

signup_months = df[['user_id', 'high_spender']].drop_duplicates().copy()
signup_months['signup_month'] = None

signup_months.loc[signup_months['high_spender'] == 0, 'signup_month'] = np.random.choice(
    np.arange(1, num_months),
    (signup_months['high_spender'] == 0).sum()
) * np.random.binomial(1, .05, size=(signup_months['high_spender'] == 0).sum())

signup_months.loc[signup_months['high_spender'] == 1, 'signup_month'] = np.random.choice(
    np.arange(1, num_months),
    (signup_months['high_spender'] == 1).sum()
) * np.random.binomial(1, .95, size=(signup_months['high_spender'] == 1).sum())

df = df.merge(signup_months)

df["treatment"] = df["signup_month"] > 0

after_signup = (df["signup_month"] < df["month"]) & (df["treatment"])
df.loc[after_signup, "spend"] = df[after_signup]["spend"] * 1.5

df

Now imagine you weren't aware of causality, confounders, high / low spenders, and all that. You simply published your rewards signup page and observed your customers' spending behaviour over a span of a year. Chances are you'll compute the average treatment effect the exact same way we did above for our randomized controlled trial:

month = 6

post_signup_spend = (
    df[df.signup_month.isin([0, month])]
    .groupby(["user_id", "signup_month", "treatment"])
    .apply(
        lambda x: pd.Series(
            {
                "post_spend": x.loc[x.month > month, "spend"].mean(),
                "pre_spend": x.loc[x.month < month, "spend"].mean(),
            }
        )
    )
    .reset_index()
)
print(post_signup_spend)

      user_id  signup_month  treatment  post_spend  pre_spend
0           1             0      False  251.666667      239.0
1           3             0      False  246.166667      252.8
2           4             0      False  740.833333      737.2
3           6             0      False  254.333333      247.0
4           7             0      False  249.166667      253.2
...       ...           ...        ...         ...        ...
5499     9992             0      False  246.000000      240.6
5500     9995             0      False  254.833333      256.4
5501     9996             0      False  248.833333      239.8
5502     9997             0      False  249.500000      247.8
5503     9999             0      False  761.666667      752.6

[5504 rows x 5 columns]

post_spend = post_signup_spend\
  .groupby('treatment')\
  .agg({'post_spend': 'mean'})

post_spend

post_spend.loc[True, 'post_spend'] - post_spend.loc[False, 'post_spend']

799.6519394104552

post_spend.loc[True, 'post_spend'] / post_spend.loc[False, 'post_spend']

3.8984263250854134

Performing the exact same computation as above, now we're estimating an average treatment effect of almost 400% instead of the actual 50%!

So what went wrong here?

Observational data got us!

Realize that in our observational data the outcome (spend) is not indepndent of treatment assignmnet (rewards program signup): High spenders are far more likely to sign up hence are overrepresented in our treatment group while low spenders are overrepresented in our control group (users that didn't sign up).

So when we compute the above difference or ratio we don't just see the average treatment effect of rewards signup we also see the inherent difference in spending between high and low spenders.

So if we ignore how our observational data are generated we'll overestimate the effect our rewards program has and likely make decisions that seem to be supported by data but in reality aren't.

Also notice that we often make this same mistake when training machine learning algorithms on observational data. Chances are someone will ask you to train a regression model to predict the effectiveness of the rewards program and your model will end up with the same inflated estimate as above.

So how do we fix this? And how can we estimate the true treatment effect from our observational data?

Generally, we know from experience in e-commerece that people who tend to spend more are more likely to sign up to our rewards program. So we could segment our users into spend buckets and compute the treatment effect within each bucket to try and breeak this confounding link in our observational data.

Notice that in practice we won't have a high spender flag for our customers so we'll have to go by our customers' observed spending behaviour.

The causal inference framework offers an established approach here: Relying on our domain knowledge, we define a causal model that describes how we believe our observational data were generated.

Let's draw this as a graph with nodes and edges:

import os, sys
sys.path.append(os.path.abspath("../../../"))
import dowhy

causal_graph = """digraph {
treatment[label="Program Signup in month i"];
pre_spend;
post_spend;
U[label="Unobserved Confounders"]; 
pre_spend -> treatment;
pre_spend -> post_spend;
treatment->post_spend;
U->treatment; U->pre_spend; U->post_spend;
}"""

model = dowhy.CausalModel(
    data=post_signup_spend,
    graph=causal_graph.replace("\n", " "),
    treatment="treatment",
    outcome="post_spend"
)
model.view_model()

Our causal model states what we described above: Pre-signup spend influences both rewards signup (treatment assignment) and post-signup spend. This is the story about our high and low spenders.

Treatment (rewards signup) influences post-signup spending behaviour - this is the effect we're actually interested in.

We also added a node U to signify possible other confounding factors that may exist in reality but weren't observed as part of our data.

Identification / identifying the causal effect

We will now apply do-calculus to our causal model from above to figure out ways to cleanly estimate the treatment effect we're after:

identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)
print(identified_estimand)

Estimand type: nonparametric-ate

### Estimand : 1
Estimand name: backdoor
Estimand expression:
     d                                         
────────────(Expectation(post_spend|pre_spend))
d[treatment]                                   
Estimand assumption 1, Unconfoundedness: If U→{treatment} and U→post_spend then P(post_spend|treatment,pre_spend,U) = P(post_spend|treatment,pre_spend)

### Estimand : 2
Estimand name: iv
No such variable found!

### Estimand : 3
Estimand name: frontdoor
No such variable found!

Very broadly and sloppily stated there a three ways to segment (or slice and dice) our observational data to get to subsets of our data within which we can cleanly compute the average treatment effect:

Backdoor adjustment,
Frontdoor adjustment, and
Instrumental variables.

The above printout tells us that based on both the causal model we constructed and our observational data there is a backdoor-adjusted estimator for our desired treatment effect.

This backdoor adjustment actually follows closely what we already said above: We'll compute the post-spend given pre-spend (segment our customers based on their spending behaviour).

Estimating the treatment effect

There are various ways to perform the backdoor adjustment that DoWhy identified for us above. One of them is called propensity score matching:

estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor1.propensity_score_matching",
    target_units="ate"
)

print(estimate)

/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:760: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)

*** Causal Estimate ***

## Identified estimand
Estimand type: nonparametric-ate

## Realized estimand
b: post_spend~treatment+pre_spend
Target units: ate

## Estimate
Mean value: 159.15597747093042

DoWhy provides us with an estimated ATE for our observational data that is pretty close to the ATE we computed for our experimental data from our randomized controlled trial.

Even if the ATE estimate DoWhy provides doesn't match exactly our experimental ATE we're now in a much better position to take a decision regarding our rewards program based on our observational data.

So next time we want to base decisions on observational data it'll be worthwhile defining a causal model of the underlying data-generating process and using a library such as DoWhy that helps us identify and apply adjustment strategies.

import pandas as pd
import numpy as np
import random

np.random.seed(123)
random.seed(123)

no_samples = 10000

seasons = np.random.choice(['winter', 'spring', 'summer', 'fall'], size=(no_samples,))
color = np.array(
    [
        random.choice(['yellow', 'pink'])
        if season in ['spring', 'summer']
        else random.choice(['navy', 'grey'])
        for season in seasons
    ]
)
price = np.random.lognormal(size=(no_samples,))
rank = np.array(
    [
        
    ]
)

seasons

array(['fall', 'spring', 'spring', ..., 'spring', 'spring', 'spring'],
      dtype='<U6')

The need for causality modelling

Intelligent planning and decision-making lie at the heart of most business success.

The decisions that our business needs to evaluate can range from those that are relatively low effort and we take potentially thousands or millions of times a day to those that are high effort and are taken every couple of months:

What will happen if I show an advertising banner to a particular user?
What will happen if I change the retail prices for certain products in my shop?
What will happen if I alter my manufacturing process?
What will happen if I swap out a particular mechanical piece in a vehicle I develop?
What will happen if I invest in new property, machinery, or processes?
What will happen if I hire this applicant?
What if I increase remuneration of my workforce?

As industrial data scientists we are oftentimes called upon to evaluate these proposed business decisions using analytics, machine learning methodologies, and past data.

What we may end up doing for the above proposed business decisions is:

Compute and rank past click-through rates for given pairs of ad banner and user,
Correlate past demand with set retail prices for product groups of interest,
Correlate past manufacturing parameters with achieved output quality,
Correlate the mechanical behavior of my vehicles with the mechnical parts used in it,
Use past data to forecast the development of real estate prices,
Use past data to correlate and predict the productivity of my team given e.g. its size or makeup, and
Use past data to correlate productivity and remuneration levels.

The way I formulated these is already pretty suggestive - but essentially some of our common approaches to evaluating business decisions do not compare our business outcomes with and without said business decisions but they rather look at our data outside the context of decision-making.

Put another way, we oftentimes analyze past data without considering the state our business or customer is in when those data were generated. For illustration:

So really when tasked with evaluating the above proposed business decisions we should instead think in terms of questions akin the following:

How would the user of interest behave differently if we didn't show them (and pay for) a banner now?
For each Euro we shave off a price tag how much higher will our revenue be since more customers are inclinced to place an order?
4. 5. 6. 7.

How to do causality modelling

The authors Hünermund and Bareinboim (https://arxiv.org/abs/1912.09104) proposed a methodology they called data-fusion process.

The data-fusion process maps out the individual steps necessary for evaluating the impact of past and potential future decisions:

Use case: The impact of direct marketing on customer behavior

We'll use a data set provided by UCI (https://archive.ics.uci.edu/ml/datasets/Bank+Marketing) that demonstrates the potential impact of direct marketing on customer success.

Let's dive right in, download the data set and see what we are working with.

The direct marketing success data set

!wget --quiet https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank.zip

!unzip -oqq bank.zip

import pandas as pd

df = pd.read_csv('bank.csv', delimiter=';')
df['success'] = df['y']
del df['y']
df['success'] = df['success'].replace('no', 0)
df['success'] = df['success'].replace('yes', 1)
del df['duration']
df['no_contacts'] = df['campaign']
del df['campaign']

df.head()

Our tabular marketing and sales data shows a number of features we observe about a given customer and our interaction with them:

The customer's age, job, marital status, education, current account balance, and whether or not they already took out a loan are recorded,
Our direct marketing interaction with a given customer is also recorded, for instance, how often we already contacted them.

A more detailed description of the features in our data can be found here:

https://archive.ics.uci.edu/ml/datasets/Bank+Marketing

Trying to help our business with machine learning only

target = 'success'
features = [column for column in df.columns if column != target]

import lightgbm as lgb
from sklearn.preprocessing import OrdinalEncoder

model = lgb.LGBMClassifier()

X, y = df[features], df[target]

numerical_features = ['age', 'balance', 'no_contacts', 'previous', 'pdays']
categorical_features = [feature for feature in features if feature not in numerical_features]

encoder = OrdinalEncoder(dtype=int)

X_numeric = pd.concat(
    [
        X[numerical_features],
        pd.DataFrame(
            data=encoder.fit_transform(X[categorical_features]),
            columns=categorical_features
        )
    ],
    axis=1
)

X_numeric.head()

model.fit(X_numeric, y)

LGBMClassifier()

%matplotlib inline

lgb.plot_importance(model);

There are numerous ways to compute feature importance and this one implemented in the LightGBM library measures the number of times a given feature is used in the constructed trees:

https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.plot_importance.html

In general, feature importance gives us a measure of how well a given measured variable correlates with the target (marketing success in our case).

The question here is: How can we use our trained success predictor and our feature importances to aid intelligent plannning and decision-making in our business?

Uses for

Summary

Marketing is a key success and revenue driver in B2C markets: An appropriate message placed at the appropriate time with a prospective customer will increase your business success.

However, marketing is also a major cost driver for businesses: Marketing efforts that are too broad, target the wrong audience, or convey the wrong message waste resources.

In the case of direct marketing via phone conversations a key cost factor is the amount of time a sales call agent spends with the prospective customer on the phone.

In this article we explore, rudimentarily, direct marketing data of a Portuguese financial institution.

We explore the relationship between call duration and success (purchase of offered financial product), and show that consideration of customer-specific factors influences how you should allocate your marketing resources.

Our prototypical analysis can be usueful in devising data-driven marketing and sales strategies that offer decision intelligence for your call agents.

Fetch the data

For our prototype we use the openly accessible Bank Marketing Data Set from the UC Irvine Machine Learning Repository.

!wget --quiet https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank.zip

!unzip -oqq bank.zip

Load Python libraries

import graphviz
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.cluster import KMeans
from sklearn.preprocessing import LabelEncoder

np.random.seed(42)

Prepare data

The data we work with here contain a number of categorical and numerical variables. To keep our analysis and prototype simple we will focus on only a handful and remove the remainder.

#collapse
df = pd.read_csv('bank.csv', delimiter=';')
df['success'] = df['y']
del df['y']
df['success'] = df['success'].replace('no', 0)
df['success'] = df['success'].replace('yes', 1)
del df['education']
del df['default']
del df['housing']
del df['loan']
del df['contact']
del df['day']
del df['month']
del df['campaign']
del df['pdays']
del df['previous']
del df['poutcome']

Our tabular data set now looks as follows: Each prospective (and in some cases eventual) customer whom a call agent conversed with fills a row. On each row we have numerical variables (age, account balance, duration of sales interaction) and categorical variables (job / employment status and marital status). Our data set contains 4,521 sales interactions.

df

High-level model: more is better

A blanket approach to marketing and sales may be: More resources lead to greater success.

So in the case of direct marketing on the phone we could expect that the more time we spend with a prospective customer on the phone, the bigger our success rate.

To test our model, we discretize the duration of our interaction with the customer into six duration buckets: bucket 1 holds the shortest interactions while bucket 6 holds the longest interactions.

no_buckets = 6
df['duration_bucket'] = pd.qcut(df['duration'], no_buckets, labels=[f'bucket {b + 1}' for b in range(no_buckets)])

df.groupby('duration_bucket').agg({'success': 'mean'})

Looking at the average success rate in each duration bucket shows us that there is positive correlation between the duration of a sales interaction and our success rate - just as our model predicted.

Hence, more marketing spend appears to lead to greater success in general.

From a data perspective this is a pretty disappointing result as we expect to glean more intelligent insights from all the data we collected.

Nuanced model: more isn't always better and there are always tradeoffs

Let's dig deeper into what is going on here: Yes, the duration of the interaction between call agent and prospective customer likely influences our success rate.

However, call agents also probably choose to spend more time on the phone with customers whose account balance is higher - hoping for a greater chance of a sale. That same account balance also likely influences how affine the customer is for spending more money on financial products.

Both present job status and marital status are also likely candidates for influencing an affinity for financial products.

And age of the customer probably influences both their job and marital status.

Since a customer's account balance probably influences both how much time we spend with them on the phone and their likelihood of purchasing another financial product we will control for account balance.

We control for account balance by training a cluster algorithm that segments our data set into three groups of similar account balance.

df['job'] = LabelEncoder().fit_transform(df['job'])
df['marital'] = LabelEncoder().fit_transform(df['marital'])

segmenter = KMeans(n_clusters=3, random_state=42)

df['segment'] = segmenter.fit_predict(df[['balance']])

df['segment'] = df['segment'].replace({0: 'low balance', 1: 'high balance', 2: 'medium balance'})

Looking at both the average account balance and age in our three segments, we notice that our clustering algorithm picked out low, medium, and high balance segments.

We also notice that average age correlates with average balance in these three segments hence our intuition codified in our above model seems valid.

df.groupby('segment').agg({'age': 'mean', 'balance': 'mean'})

Now, what about the effectiveness of our marketing resources in each segment?

Visualizing our rate of success in the six duration buckets broken down by account balance segment we see a more nuanced picture:

Customers with low account balances really need to be worked on and only show success rates greater than 20% in the highest duration bucket 6,
customers with medium balances already show a greater than 20% purchase likelihood in duration bucket 4, and
customers with high balances actually max out in duration bucket 5 and drop below a 20% success rate in bucket 6.

success_rates = df.groupby(['segment', 'duration_bucket']).agg({'success': 'mean'}).reset_index()

sns.set(rc={'figure.figsize': (10,6)})
sns.barplot(
    x='duration_bucket',
    y='success',
    hue='segment',
    data=success_rates,
    hue_order=['low balance', 'medium balance', 'high balance']
);

Our more nuanced model and analysis provide us with data-driven insights that provide actionable and testable advice:

We should probably re-evaluate whether low balance individuals are sensible targets for our marketing campaigns given how resource-intensive they are,
compute the profit and loss tradeoff between spending bucket 4 and bucket 6 resources on medium balance individuals, and
ensure that we do not overdo it with our calls for high balance individuals.

Summary

Equipment that behaves anomalously or breaks down unexpectedly is a major cost driver in manufacturing, logistics, public transport, and any other sector that relies on complex machinery.

A big promise of data analytics and machine learning in this space is to detect anomalies in machinery automatically and to alert their user of occurring faults. As an extension, the prediction of machinery faults and breakdowns is an important field of application.

Automated detection and prediction of machinery breakdown is a key algorithmic approach behind smart and predictive maintenance.

In this article we showcase a simple algorithmic approach for anomaly detection in the space of automated engine health detection.

Our approach here can be an interesting starting point for the development of smart telematics solutions for automated and predictive vehicle breakdown detection.

Fetch the data

We'll make use of an open data set of vehicle CAN bus data, called Automotive CAN bus data: An Example Dataset from the AEGIS Big Data Project.

A CAN bus is a local network of sensors and actuators in modern vehicles that provides a stream of data for all important signals of a vehicle - such as its present velocity, interior temperature, and potentially hundreds of other signals.

This data set encompasses time series data (traces) of various vehicles driven by different drivers.

Let's go ahead and download a data set for driver 1 and a data set for driver 2:

!wget --quiet https://zenodo.org/record/3267184/files/20181113_Driver1_Trip1.hdf

!wget --quiet https://zenodo.org/record/3267184/files/20181114_Driver2_Trip3.hdf

Load libraries

Here we import all necessary Python libraries for our analysis and algorithm:

import h5py
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

from sklearn.mixture import GaussianMixture

plt.rcParams['figure.figsize'] = (10,10)
sns.set(style="darkgrid")

Load vehicle data

Let's load the data for driver 1 and driver 2 into memory:

driver_1 = h5py.File('20181113_Driver1_Trip1.hdf', 'r')
driver_2 = h5py.File('20181114_Driver2_Trip3.hdf', 'r')

Both files contain multiple subgroups of data, one of which is the aformentioned CAN bus:

list(driver_1.keys())

['AI', 'CAN', 'GPS', 'Math', 'Plugins']

list(driver_2.keys())

['AI', 'CAN', 'GPS', 'Math', 'Plugins']

Turn time series data into tables

The CAN bus data comes in serialized form - written out in series in a nested format.

To handle the CAN bus data more efficiently we'll turn it into tables that are easier to inspect and handle.

data_driver_1 = {}
data_driver_2 = {}

for channel_name, channel_data in driver_1['CAN'].items():
  data_driver_1[channel_name] = channel_data[:, 0]

table_driver_1 = pd.DataFrame(
    data=data_driver_1,
    index=channel_data[:, 1]
)
table_driver_1 = table_driver_1.loc[:, table_driver_1.nunique() > 1]

for channel_name, channel_data in driver_2['CAN'].items():
  data_driver_2[channel_name] = channel_data[:, 0]

table_driver_2 = pd.DataFrame(
    data=data_driver_2,
    index=channel_data[:, 1]
)
table_driver_2 = table_driver_2.loc[:, table_driver_2.nunique() > 1]

The tabular data for driver 1 looks as follows - it holds 158,659 measured time points in 28 channels that we deem relevant:

table_driver_1

The tabular data for driver 2 looks as follows - it holds 136,154 measured time points in 29 channels that we deem relevant:

table_driver_2

Monitoring engine health

One use case of automated anomaly detection lies in checking the health status of a vehicle's engine.

Here we'll look at engine oil temperature as a function of velocity: As you'll notice in the below plots, oil temperature goes up with higher velocity and increased operating duration.

Engine 1 - healthy

Let's look at the engine of the vehicle of driver 1 where engine oil temperature appears normal as it keeps to within a certain band:

temperature_1 = table_driver_1[['VehicleSpeed', 'OilTemperature1']].copy()

sns.relplot(
    x='index',
    y='value',
    hue='channel',
    kind='line',
    data=temperature_1.reset_index().melt(id_vars='index', var_name='channel')
);

Plotting engine oil temperature against vehicle velocity makes the correlation between the two metrics more apparent:

plot = sns.scatterplot(
    x='VehicleSpeed',
    y='OilTemperature1',
    data=temperature_1,
    color='blue',
    alpha=.1
);
plot.axes.set_ylim(0., 110.);

Engine 2 - unhealthy

Let's look at velocity and engine oil temperature for vehicle 2 (where I deliberately introduce an anomaly between 5000 and 5500 seconds of the trace).

You'll notice a spike in engine oil temperature which indicates unhealthy behavior of the vehicle's engine:

temperature_2 = table_driver_2[['VehicleSpeed', 'OilTemperature1']].copy()
temperature_2.loc[5000:5500, 'OilTemperature1'] *= 1.15

sns.relplot(
    x='index',
    y='value',
    hue='channel',
    kind='line',
    data=temperature_2.reset_index().melt(id_vars='index', var_name='channel')
);

plot = sns.scatterplot(
    x='VehicleSpeed',
    y='OilTemperature1',
    data=temperature_2,
    color='blue',
    alpha=.1
);
plot.axes.set_ylim(0., 140.);

Anomaly detection algorithm: learn what observations to expect to recognize anomalies

Let's look at the visual distribution of engine oil temperature and velocity again and directly compare vehicle 1 with vehicle 2.

temperature_1['vehicle'] = 'vehicle 1'
temperature_2['vehicle'] = 'vehicle 2'
combined = pd.concat([temperature_1, temperature_2], axis=0, sort=True)

plot = sns.scatterplot(
    x='VehicleSpeed',
    y='OilTemperature1',
    data=combined,
    hue='vehicle',
    alpha=.1
);
plot.axes.set_ylim(0., 140.);

You'll notice that the engine oil temperature of vehicle 2 tends to be higher than of vehicle 1 and vehicle 2 shows an island of high engine oil temperature separated from the bulk of data points.

We can use this by modelling the distribution of value pairs velocity and oil temperature that we would usually expect to observe.

Let's model the expected distribution of data points with a model called Gaussian mixture models. This model fits a set of Gaussian distributions to the distribution of value pairs we observed for vehicle 1 - thus defining what a healthy distribution of values looks like.

model = GaussianMixture(n_components=4)

model.fit(temperature_1[['VehicleSpeed', 'OilTemperature1']])

GaussianMixture(covariance_type='full', init_params='kmeans', max_iter=100,
                means_init=None, n_components=4, n_init=1, precisions_init=None,
                random_state=None, reg_covar=1e-06, tol=0.001, verbose=0,
                verbose_interval=10, warm_start=False, weights_init=None)

After training our model on the observations for vehicle 1, let's score the likelihood of observing each observation we have on vehicle 2:

temperature_2['health_score'] = model.score_samples(temperature_2[['VehicleSpeed', 'OilTemperature1']])

sns.relplot(
    x='index',
    y='value',
    hue='metric',
    kind='line',
    data=temperature_2[['OilTemperature1', 'health_score']].reset_index().melt(id_vars='index', var_name='metric')
);

To clarify what we did here: We learned what a healthy distribution of vehicle velocity and engine oil temperature looks like based on data from vehicle 1 and applied this model to data from vehicle 2.

Looking at the health score we compute for vehicle 2 we notice that our health score drops markdely between 5000 and 5500 seconds into our CAN bus trace - exactly where oil temperature spikes unhealthily.

While we would need to do a lot more validation and model calibration before we could use this approach in a live environment, we can already see the potential of this approach.

With this approach we can start devising data-driven products and services for smart telematics and smart maintenance applications.

Classifying Wikipedia Changes

I recently joined a Kaggle competition on multilabel text classification and have learned a ton from basic code that one of the competitors shared in the forums.

The code of the genereous competitor does logistic regression classification for multiple classes with stochastic gradient ascent. It is further well-suited for online learning as it uses the hashing trick to one-hot encode boolean, string, and categorial features.

To better understand these methods and tricks I here apply some of them to a multilabel problem I chose mostly for the easy access to a constant stream of training data:

All recent changes on Wikipedia are tracked on this special page where we can see a number of interesting features such as the length of the change, the contributor's username, the title of the changed article, and the contributor's comment for a given change.

Using the Wikipedia API to look at this stream of changes we can also see how contributors classify their changes as bot, minor, and new. Multiple label assignments are possible, so that one contribution may be classified as both bot and new.

Here I will listen to this stream of changes, extract four features (length of change, comment string, username, and article title), and train three logistic regression classifiers (one for each class) to predict the likelihood of a change belonging to each one of them. The training is done with the stochastic gradient ascent method.

One caveat: I am a complete novice when it comes to most of this stuff so please take everything that follows with a grain of salt - on the same note I would be forever grateful for any feedback especially of the critical kind so that I can learn and improve.

%matplotlib inline
import matplotlib
from matplotlib import pyplot as pt
import requests
import json
from math import log, exp, sqrt
from datetime import datetime, timedelta
import itertools
from collections import defaultdict

The API that Wikipedia offer to listen to the stream of recent changes is described here.

URL = ('http://en.wikipedia.org/w/api.php?format=json&action=query&list=recentchanges&rcprop=parsedcomment'
      '%7Ctimestamp%7Ctitle%7Cflags%7Cids%7Csizes%7Cflags%7Cuser&rclimit=100')

The logistic regression classifier requires us to compute the dot product between a feature vector $\mathbf{x}$ and a weight vector $\mathbf{w}$.

$$\mathbf{w}^\text{T} \mathbf{x} = w_0 x_0 + w_1 x_1 + w_2 x_2 + \ldots + w_N x_N.$$

As by convention, the bias of the model is encoded with feature $x_0 = 1$ for all observations - the only thing that will change about the $w_0 x_0$-term is weight $w_0$ upon training. The length of the article change is tracked with numerical feature $x_1$ which equals the number of character changes (hence $x_1$ is either positive or negative for text addition and removal respectively).

As in the Kaggle code that our code is mostly based upon, string features are one-hot encoded using the hashing trick:

The string features extracted for each observed article change are username, a parse of the comment, and the title of article. Since this is an online learning problem there is no way of knowing how many unique usernames, comment strings, and article titles are going to be observed.

With the hashing trick we decide ab initio that D_sparse-many unique values across these three features are sufficient to care about: Our one-hot encoded feature space has dimension D_sparse and can be represented as a D_sparse-dimensional vector filled with 0's and 1's (feature not present / present respectively).

The hash in hashing trick comes from the fact that we use a hash function to convert strings to integers. Suppose now that we chose D_sparse = 3 and our hash function produces hash("georg") = 0, hash("georgwalther") = 2, and hash("walther") = 3 for three observed usernames.

For username georg we get feature vector $[1, 0, 0]$ and for username georgwalther we get $[0, 0, 1]$. The hash function maps username walther outside our 3-dimensional feature space and to close this loop we not only use the hash function but also the modulus (which defines an equivalence relation?):

hash("georg") % D_sparse = 0, hash("georgwalther") % D_sparse = 2, and hash("walther") % D_sparse = 0

This illustrates one downside of using the hashing trick since we will now map usernames georg and walther to the same feature vector $[1, 0, 0]$. We are therefore best adviced to choose a big D_sparse to avoid mapping different feature values to the same one-hot-encoded feature - but probably not too big to preserve memory.

For each article change observation we only map three string features into this D_sparse-dimensional one-hot-encoded feature space - out of D_sparse-many vector elements there will only ever be three ones among (D_sparse-3) zeros (if we do not map to the same vector index multiple times). We will therefore use sparse encoding for these feature vectors (hence the sparse in D_sparse).

We will also normalize the length of change on the fly using an online algorithm for mean and variance estimation.

D = 2  # number of non-sparse features
D_sparse = 2**18  # number of sparsely-encoded features

def get_length_statistics(length, n, mean, M2):
    """ https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Incremental_algorithm """
    n += 1
    delta = length-mean
    mean += float(delta)/n
    M2 += delta*(length - mean)
    
    if n < 2:
        return mean, 0., M2

    variance = float(M2)/(n - 1)
    std = sqrt(variance)
    return mean, std, M2

def get_data():
    X = [1., 0.]  # bias term, length of edit
    X_sparse = [0, 0, 0]  # hash of comment, hash of username, hash of title
    Y = [0, 0, 0]  # bot, minor, new

    length_n = 0
    length_mean = 0.
    length_M2 = 0.
    
    while True:
        r = requests.get(URL)
        r_json = json.loads(r.text)['query']['recentchanges']
        for el in r_json:
            length = abs(el['newlen'] - el['oldlen'])
            length_n += 1
            length_mean, length_std, length_M2 = get_length_statistics(length, length_n, length_mean, length_M2)
            X[1] = (length - length_mean)/length_std if length_std > 0. else length

            X_sparse[0] = abs(hash('comment_' + el['parsedcomment'])) % D_sparse
            X_sparse[1] = abs(hash('username_' + el['user'])) % D_sparse
            X_sparse[2] = abs(hash('title_' + el['title'])) % D_sparse

            Y[0] = 0 if el.get('bot') is None else 1
            Y[1] = 0 if el.get('minor') is None else 1
            Y[2] = 0 if el.get('new') is None else 1

            yield Y, X, X_sparse

def predict(w, w_sparse, x, x_sparse):
    """ P(y = 1 | (x, x_sparse), (w, w_sparse)) """
    wTx = 0.
    for i, val in enumerate(x):
        wTx += w[i] * val

    for i in x_sparse:
        wTx += w_sparse[i]  # *1 if i in x_sparse

    try:
        wTx = min(max(wTx, -100.), 100.)
        res = 1./(1. + exp(-wTx))
    except OverflowError:
        print wTx
        raise
        
    return res

def update(alpha, w, w_sparse, x, x_sparse, p, y):
    for i, val in enumerate(x):
        w[i] += (y - p) * alpha * val
    for i in x_sparse:
        w_sparse[i] += (y - p) * alpha  # * feature[i] but feature[i] == 1 if i in x

K = 3
w = [[0.] * D for k in range(K)]
w_sparse = [[0.] * D_sparse for k in range(K)]
predictions = [0.] * K
alpha = .1

time0 = datetime.now()
training_time = timedelta(minutes=10)

ctr = 0
for y, x, x_sparse in get_data():
    for k in range(K):
        p = predict(w[k], w_sparse[k], x, x_sparse)
        predictions[k] = float(p)
        update(alpha, w[k], w_sparse[k], x, x_sparse, p, y[k])

    ctr += 1
    
    # if ctr % 10000 == 0:
        # print 'samples seen', ctr
        # print 'sample', y
        # print 'predicted', predictions
        # print ''

    if (datetime.now() - time0) > training_time:
        break

ctr

As we can see, we crunched through 106,401 article changes during our ten-minute online training.

It would be fairly hard to understand the link between the D_sparse-dimensional one-hot-encoded feature space and the observed / predicted classes. However we can still look at the influence that the length of the article change has on our classification problem

print w[0]
print w[1]
print w[2]

Here we can see that the weight of the length of change for class 0 (bot) is -1.12, for class 1 (minor) is -0.97, and for class 2 (new) is 2.11.

Intuitively this makes sense since many added characters (big positive change) should make classification as a minor change less likely and classification as a new article more likely: For an observed positive character count change $C$, $2.11 C$ will place us further to the right, and $-0.97 C$ further to the left along the $x$-axis of the sigmoid function:

(from http://gaelvaroquaux.github.io/scikit-learn-tutorial/supervised_learning.html#classification)

Further below we will see that the vast majority of bot changes are classified as minor changes hence we would expect to see correlation between the weights of this feature for these two classes.

To further evaluate our three classifiers, we observe another 10,000 Wikipedia changes and construct a confusion matrix.

no_test = 10000
test_ctr = 0

classes = {c: c_i for c_i, c in enumerate(list(itertools.product([0, 1], repeat=3)))}
confusion_matrix = [[0 for j in range(len(classes))] for i in range(len(classes))]

predicted = [0, 0, 0]
for y, x, x_sparse in get_data():
    for k in range(K):
        p = predict(w[k], w_sparse[k], x, x_sparse)
        predicted[k] = 1 if p > .5 else 0
    i = classes[tuple(y)]
    j = classes[tuple(predicted)]
    confusion_matrix[i][j] += 1
    test_ctr +=1
    
    if test_ctr >= no_test:
        break

matplotlib.rcParams['font.size'] = 15
fig = pt.figure(figsize=(11, 11))
pt.clf()
ax = fig.add_subplot(111)
ax.set_aspect(1)
res = ax.imshow(confusion_matrix, cmap=pt.cm.jet, interpolation='nearest')
cb = fig.colorbar(res)
labels = [{v: k for k, v in classes.iteritems()}[i] for i in range(len(classes))]
pt.xticks(range(len(classes)), labels)
pt.yticks(range(len(classes)), labels)
pt.show()

The confusion matrix shows actual classes along the vertical and predicted classes along the horizontal axis.

The vast majority of observed classes are $(0, 0, 0)$ and our classifiers get most of these right except that some are misclassified as $(0, 0, 1)$ and $(0, 1, 0)$.

All observed bot-related changes (classes starting with a $1$) are $(1, 1, 0)$ (i.e. minor bot-effected changes) and our classifiers get all of those right.

PLOS Biology-Inspired PLOS Biology Articles

This past week I had my first encounter with the concept of graph databases which lend themselves perfectly to modeling and capturing linked data.

I started reading the free and brilliant book Graph Databases by Robinson, Webber, and Eifrem and began playing around with Python bulbs by James Thornton.

I further took the data set of 1754 PLOS Biology articles that I have examined on this blog multiple times and created a Rexster-based graph database from them. Apart from the obvious authors, DOIs, and titles I also extracted references to other PLOS Biology articles.

In this blog post I will examine these links between PLOS Biology articles.

Let us first take a look at my database to get an idea of what this looks like.

%matplotlib inline
from matplotlib import pyplot

from bulbs.rexster import Graph, Config, REXSTER_URI

REXSTER_URI = 'http://localhost:8182/graphs/plos'
config = Config(REXSTER_URI)

g = Graph(config)

The label g now holds a reference to our graph database.

Python bulbs allows us to define classes for our data model which is something I did when creating this graph database in the first place. These are the node (vertex) types and edge (relationship) types I defined:

# Bulbs Models
from bulbs.model import Node, Relationship
from bulbs.property import String, Integer, DateTime, List

class Author(Node):
    element_type = 'author'
    name = String(nullable=False)
    
class Article(Node):
    element_type = 'article'
    title = String(nullable=False)
    published = DateTime()
    doi = String()
    
class Authorship(Relationship):
    label = 'authored'

class Citation(Relationship):
    label = 'cites'
    reference_count = Integer(nullable=False)
    tag = String()

This is a very basic model of PLOS Biology articles that captures nothing more than authorship (edges between authors and articles) and citations (edges between articles).

Some of these concepts can and should probably be decorated further: for instance Authorship edges could include author contributions (as provided at the bottom of most PLOS Biology articles).

g.add_proxy('authors', Author)
g.add_proxy('articles', Article)
g.add_proxy('authored', Authorship)
g.add_proxy('cites', Citation)

Usually we would use Rexster/Bulbs-builtin functions that rely on some internal index but since that index seems to be broken for me right now I will simply collect all nodes and edges by hand and create Python dictionaries as indeces.

This is okay here to do since our database is very small but would likely be prohibitive for anything marginally bigger.

nodes = g.V
edges = g.E

authors = {n.name: n for n in nodes if n.element_type == 'author'}

authors.keys()[:10]

[u'Shuguang Zhang',
 u'Ernst Hafen',
 u'Maren Brockmeyer',
 u'Bruno Eschli',
 u'David B. Gurevich',
 u'Michael Lynch',
 u'Alejandro Valbuena',
 u'Claudia Rutte',
 u'Matthew M Wyatt',
 u'Brianna B. Williams']

articles = {n.doi: n for n in nodes if n.element_type == 'article'}

articles.keys()[:10]

[u'10.1371/journal.pbio.0040216',
 u'10.1371/journal.pbio.0040215',
 u'10.1371/journal.pbio.0040210',
 u'10.1371/journal.pbio.0040368',
 u'10.1371/journal.pbio.0040369',
 u'10.1371/journal.pbio.0040362',
 u'10.1371/journal.pbio.0040363',
 u'10.1371/journal.pbio.0040360',
 u'10.1371/journal.pbio.0020275',
 u'10.1371/journal.pbio.0040366']

Let us now do a brief sanity check and count the number of PLOS Biology articles in our data set (this should equal 1754).

len(articles.keys())

1754

Let us now pick an article at random and see how this article is connected to the remainder of the graph.

article = articles['10.1371/journal.pbio.1000584']

This is the title of article:

article.title

u'Clusters of Temporal Discordances Reveal Distinct Embryonic Patterning Mechanisms in Drosophila and Anopheles'

These are the edges pointing to this article:

list(article.inE())

[<Authorship: http://localhost:8182/graphs/plos/edges/21895>,
 <Authorship: http://localhost:8182/graphs/plos/edges/21893>,
 <Authorship: http://localhost:8182/graphs/plos/edges/21891>]

There are three Authorship edges that point to this specific article.

To get the node at the base of a directed edge we can either query article.inE().inV() (i.e. the in-node of this edge) or simply ask for the in-node of the article node straight away - this should be equivalent!

for author in article.inV():
    print author.name

Yury Goltsev
Michael Levine
Dmitri Papatsenko

A quick check online confirms that these are indeed the authors of article.

As I mentioned above, I also collected all references to other PLOS Biology articles in my data set and modeled those as Citation relationships (edges) between articles.

The article we are currently looking at has one such out-edge to another PLOS Biology article:

list(article.outE())

[<Citation: http://localhost:8182/graphs/plos/edges/29290>]

for citation in article.outV():
    print citation.title
    print [n.name for n in citation.inV() if n.element_type == 'author']
    print citation.doi

The Cell Cycle–Regulated Genes of Schizosaccharomyces pombe
[u'Saumyadipta Pyne', u'Janet Leatherwood', u'Anna Oliva', u'Bruce Futcher', u'Adam Rosebrock', u'Steve Skiena', u'Francisco Ferrezuelo', u'Haiying Chen']
10.1371/journal.pbio.0030225

As you can see above, querying our database for the authors of the PLOS Biology article that our current article (article) cites is simple.

How many PLOS Biology articles in our data set of 1754 articles cite other PLOS Biology articles?

(caveat: this only represents those citations that I detected when parsing my set of articles)

sum(1 for n in nodes if n.element_type == 'article' and n.outV() > 0)

526

I did not only extract citation edges between PLOS Biology articles but also counted how often such a citation occurs in the body of the article.

For our article and its one cited PLOS Biology article I counted:

for citation in article.outE():
    print citation.reference_count

2

Just to verify this, look up article online (DOI = 10.1371/journal.pbio.0030225) and look for reference [4] which corresponds to this one cited PLOS Biology article.

Let us now take a look at the observed distribution of how often cited PLOS Biology articles are referenced in the main text of the citing PLOS Biology article.

citation_counts = []
for doi in articles.keys():
    if articles[doi].outE():
        for e in articles[doi].outE():
            if e.label == 'cites':
                citation_counts.append(e.reference_count)

pyplot.hist(citation_counts, bins=range(20))
pyplot.xlabel('number of times cited')
pyplot.ylabel('count')

<matplotlib.text.Text at 0x9a293d0>

This histogram has a surprisingly long tail. Let us take a look at some of the bigger values to see if these make sense.

def article_pp(article):
    authors = unicode(', '.join([n.name for n in article.inV() if n.element_type == 'author']))
    s = ('Title: %s\n'
        'Authors: %s\n'
        'DOI: %s' % (article.title, authors, article.doi))
    
    return s

for edge in edges:
    if edge.label == 'cites':
        if edge.reference_count >= 21:
            print('Citer:')
            print(article_pp(edge.outV()))
            print('')
            print('Citee')
            print(article_pp(edge.inV()))
            print('')
            print('Citer cites citee %d times.' % edge.reference_count)
            print('-----------------------------------------------------')

Citer:
Title: A Feedback Loop between Dynamin and Actin Recruitment during Clathrin-Mediated Endocytosis
Authors: Marko Lampe, Christien J. Merrifield, Marcus J. Taylor
DOI: 10.1371/journal.pbio.1001302

Citee
Title: A High Precision Survey of the Molecular Dynamics of Mammalian Clathrin-Mediated Endocytosis
Authors: Marcus J. Taylor, David Perrais, Christien J. Merrifield
DOI: 10.1371/journal.pbio.1000604

Citer cites citee 21 times.
-----------------------------------------------------
Citer:
Title: H2A.Z-Mediated Localization of Genes at the Nuclear Periphery Confers Epigenetic Memory of Previous Transcriptional State 
Authors: Yvonne Fondufe-Mittendorf, Sara Ahmed, Jason H Brickner, Donna Garvey Brickner, Jonathan Widom, Ivelisse Cajigas, Pei-Chih Lee
DOI: 10.1371/journal.pbio.0050081

Citee
Title: Gene Recruitment of the Activated INO1 Locus to the Nuclear Membrane
Authors: Peter Walter, Jason H Brickner
DOI: 10.1371/journal.pbio.0020342

Citer cites citee 21 times.
-----------------------------------------------------

Checking these by hand we verify that our counts are correct.

I think it is sensible to postulate that the more often one article cites another one, the more heavily the work presented in the citer was influenced by the citee.

There is certainly some cut-off at which importance stops increasing - my point is simply that citing another article multiple times in your manuscript probably means that you are basing your work at least partially on the article you cite.

In the above list we can already see that one article titled A sex-ratio Meiotic Drive System in Drosophila simulans. II: An X-linked Distorter is a clear follow-up to the article titled A sex-ratio Meiotic Drive System in Drosophila simulans. I: An Autosomal Suppressor.

One question I am interested in is: How inspired are authors by their own work (generally very inspired I would presume), and how inspiring are articles to a completely different group of authors?

In my opinion, if one group of authors inspires a completely different group of authors to carry out scientific work (be it to follw up, refute, or whatever) then that defines knowledge transfer and a point at which scientific knowledge really becomes worth the time and resources it cost to produce this knowledge in the first place.

(I am certain this statement can be refined further but roughly speaking this is what I think)

Let us redo the above histogram but exclude all cited PLOS Biology articles that have one or more authors in common with the citing article.

(one more bracketed caveat: When constructing my database I assumed that every author name occurs exactly once and is therefore unique - this is a heuristic that breaks easily)

def are_different_authors(article_1, article_2):
    authors_1 = []
    authors_2 = []
    
    for n in article_1.inV():
        if n.element_type == 'author':
            authors_1.append(n.name)
    for n in article_2.inV():
        if n.element_type == 'author':
            authors_2.append(n.name)
            
    authors_1 = set(authors_1)
    authors_2 = set(authors_2)
    
    return len(authors_1.intersection(authors_2)) == 0

citation_counts = []

for edge in edges:
    if edge.label == 'cites':
        if are_different_authors(edge.inV(), edge.outV()):
            citation_counts.append(edge.reference_count)

pyplot.hist(citation_counts, bins=range(20))
pyplot.xlabel('number of times cited')
pyplot.ylabel('count')

<matplotlib.text.Text at 0xaa25310>

This histogram does not look very different from the one above.

Let us take a look at data points in the tail:

for edge in edges:
    if edge.label == 'cites':
        if edge.reference_count >= 16 and are_different_authors(edge.inV(), edge.outV()):
            print('Citer:')
            print(article_pp(edge.outV()))
            print('')
            print('Citee')
            print(article_pp(edge.inV()))
            print('')
            print('Citer cites citee %d times.' % edge.reference_count)
            print('-----------------------------------------------------')

Citer:
Title: Lack of Support for the Association between GAD2 Polymorphisms and Severe Human Obesity
Authors: Frank Geller, John P Kane, Raphael Merriman, Christian Vaisse, Winfried Rief, Robert Dent, Johannes Hebebrand, Björn Waldenmaier, Franck Mauvais-Jarvis, Anke Hinney, Michael M Swarbrick, Clive R Pullinger, Mary Malloy, Len A Pennacchio, Anna Ustaszewska, Denise L Lind, Wen-Chi Hsueh, Ruth McPherson, Martha M Cavazos, André Scherag, Pui-Yan Kwok
DOI: 10.1371/journal.pbio.0030315

Citee
Title: GAD2 on Chromosome 10p12 Is a Candidate Gene for Human Obesity
Authors: Lynn Bekris, Valérie Vasseur-Delannoy, Philippe Boutin, Karin Séron, Philippe Froguel, Mohamed Chikri, Christian Dina, Laetitia Corset, M. Aline Charles, Séverine Dubois, Francis Vasseur, Janice Cabellon, Ake Lernmark, Bernadette Neve, Karine Clement
DOI: 10.1371/journal.pbio.0000068

Citer cites citee 17 times.
-----------------------------------------------------
Citer:
Title: Structural Basis of Rap Phosphatase Inhibition by Phr Peptides
Authors: Alberto Marina, Francisca Gallego del Sol
DOI: 10.1371/journal.pbio.1001511

Citee
Title: Structural Basis of Response Regulator Inhibition by a Bacterial Anti-Activator Protein
Authors: Matthew B. Neiditch, Melinda D. Baker
DOI: 10.1371/journal.pbio.1001226

Citer cites citee 16 times.
-----------------------------------------------------

As we can see the two article pairs in the tail of this updated distribution are linked with lower reference counts than what we observed before filtering for author disjointedness.

Now, how inspiring are PLOS Biology authors for other (different) PLOS Biology authors?

To answer this question, I would like to propose a measure that I have called Inspiration Factor in my own head for some time now and one variant of the model I have had in mind is this:

Inspiration is an increasing function of the number of authors (unrelated to you) that you inspired to carry out scientific work.

Since I do not want to count citations that are mentioned only once in the main text of an article, I will impose a threshold of at least three references.

I should refine the way I parse articles to account for the context that citations are referenced in.

Anyways, let us take a look at those PLOS Biology articles that have inspired at least three other PLOS Biology articles.

inspirators = []

for article in articles.values():
    in_nodes = []
    if article.inE():
        for edge in article.inE():
            if edge.label == 'cites':
                if are_different_authors(edge.inV(), edge.outV()) and edge.reference_count >= 3:
                    in_nodes.append([edge.outV(), edge.reference_count])
                    
    if len(in_nodes) >= 3:
        inspirators.append([article, in_nodes])

len(inspirators)

2

for inspirator in inspirators:
    print('Inspirator')
    print article_pp(inspirator[0])
    print('')
    for el in inspirator[1]:
        print('Inspired Article')
        print article_pp(el[0])
        print('Cites inspirator %d times.' % el[1])
        print('')
    print('--------------------------------------')
    print('')

Inspirator
Title: The Evolution of Combinatorial Gene Regulation in Fungi
Authors: Alexander D Johnson, Aaron D Hernday, Hao Li, Brian B Tuch, David J Galgoczy
DOI: 10.1371/journal.pbio.0060038

Inspired Article
Title: Biofilm Matrix Regulation by Candida albicans Zap1
Authors: Oliver R. Homann, Clarissa J. Nobile, Jean-Sebastien Deneault, Aaron P. Mitchell, Andre Nantel, Aaron D. Hernday, David R. Andes, Jeniel E. Nett, Alexander D. Johnson
DOI: 10.1371/journal.pbio.1000133
Cites inspirator 3 times.

Inspired Article
Title: Evolutionary Tinkering with Conserved Components of a Transcriptional Regulatory Network
Authors: Jaideep Mallick, Adnane Sellam, Hugo Lavoie, Hervé Hogues, Malcolm Whiteway, André Nantel
DOI: 10.1371/journal.pbio.1000329
Cites inspirator 6 times.

Inspired Article
Title: Evolution of Phosphoregulation: Comparison of Phosphorylation Patterns across Yeast Species
Authors: Assen Roguev, Dorothea Fiedler, Jonathan C. Trinidad, Wendell A. Lim, Pedro Beltrao, Kevan M. Shokat, Alma L. Burlingame, Nevan J. Krogan
DOI: 10.1371/journal.pbio.1000134
Cites inspirator 3 times.

--------------------------------------

Inspirator
Title: Transcription Factors Bind Thousands of Active and Inactive Regions in the Drosophila Blastoderm 
Authors: Lisa Simirenko, Michael B Eisen, Mark Stapleton, Richard Weiszmann, Cris L. Luengo Hendriks, Tom Gingeras, Amy Beaton, Hou Cheng Chu, Xiao-yong Li, Terence P Speed, Victor Sementchenko, Mark D Biggin, Richard Bourgon, Stewart MacArthur, William Inwood, Susan E Celniker, Nobuo Ogawa, Venky N Iyer, David W Knowles, Daniel A Pollard, David Nix, Aaron Hechmer
DOI: 10.1371/journal.pbio.0060027

Inspired Article
Title: Target Genes of the MADS Transcription Factor SEPALLATA3: Integration of Developmental and Hormonal Pathways in the Arabidopsis Flower
Authors: Cezary Smaczniak, Kerstin Kaufmann, Pawel Krajewski, Ruy Jauregui, Chiara A Airoldi, Gerco C Angenent, Jose M Muiño
DOI: 10.1371/journal.pbio.1000090
Cites inspirator 3 times.

Inspired Article
Title: Evolutionary Plasticity of Polycomb/Trithorax Response Elements in Drosophila Species
Authors: Arne Hauenschild, Leonie Ringrose, Renato Paro, Christina Altmutter, Marc Rehmsmeier
DOI: 10.1371/journal.pbio.0060261
Cites inspirator 6 times.

Inspired Article
Title: Quantitative Analysis of the Drosophila Segmentation Regulatory Network Using Pattern Generating Potentials
Authors: Sudhir Kumar, Susan E. Celniker, Ann S. Hammonds, Saurabh Sinha, Majid Kazemian, Charles Blatti, Noriko Wakabayashi-Ito, Scot A. Wolfe, Adam Richards, Michael McCutchan, Michael H. Brodsky
DOI: 10.1371/journal.pbio.1000456
Cites inspirator 7 times.

--------------------------------------

And that is it for now.

I will expand my dataset to include more articles and think about how to enrich the data I extract from these articles.

One question that I am very intrigued to tackle soon is: How long of a chain of scientific discovery do you trigger?

I imagine that an article that lies at the beginning of a long chain of articles that inspired one another would have some significance.

PLOS Biology Topics

Ever wonder what topics are discussed in PLOS Biology articles? Here I will apply an implementation of Latent Dirichlet Allocation (LDA) on a set of 1,754 PLOS Biology articles to work out what a possible collection of underlying topics could be.

I first read about LDA in Building Machine Learning Systems with Python co-authored by Luis Coelho.

LDA seems to have been first described by Blei et al. and I will use the implementation provided by gensim which was written by Radim Řehůřek.

import gensim

import plospy
import os

import nltk

import cPickle as pickle

With the following lines of code we open, parse, and tokenize all 1,754 PLOS Biology articles in our collection.

As this takes a bit of time and memory, I carried out all of these steps once and stored the resulting data structures to my hard disk for later reuse - see further below.

all_names = [name for name in os.listdir('../plos/plos_biology/plos_biology_data') if '.dat' in name]

article_bodies = []

for name_i, name in enumerate(all_names):
    docs = plospy.PlosXml('../plos/plos_biology/plos_biology_data/'+name)
    for article in docs.docs:
        article_bodies.append(article['body'])

We have 1,754 PLOS Biology articles in our collection:

len(article_bodies)

punkt_param = nltk.tokenize.punkt.PunktParameters()
punkt_param.abbrev_types = set(['et al', 'i.e', 'e.g', 'ref', 'c.f',
                                'fig', 'Fig', 'Eq', 'eq', 'eqn', 'Eqn',
                                'dr', 'Dr'])
sentence_splitter = nltk.tokenize.punkt.PunktSentenceTokenizer(punkt_param)

sentences = []
for body in article_bodies:
    sentences.append(sentence_splitter.tokenize(body))

articles = []
for body in sentences:
    this_article = []
    for sentence in body:
        this_article.append(nltk.tokenize.word_tokenize(sentence))
    articles.append(this_article)

pickle.dump(articles, open('plos_biology_articles_tokenized.list', 'w'))

articles = pickle.load(open('plos_biology_articles_tokenized.list', 'r'))

is_stopword = lambda w: len(w) < 4 or w in nltk.corpus.stopwords.words('english')

Save each article as one list of tokens and filter out stopwords:

articles_unfurled = []
for article in articles:
    this_article = []
    for sentence in article:
        this_article += [token.lower().encode('utf-8') for token in sentence if not is_stopword(token)]
    articles_unfurled.append(this_article)

pickle.dump(articles_unfurled, open('plos_biology_articles_unfurled.list', 'w'))

articles_unfurled = pickle.load(open('plos_biology_articles_unfurled.list', 'r'))

Dictionary and Corpus Creation

Create a dictionary of all words (tokens) that appear in our collection of PLOS Biology articles and create a bag of words object for each article (doc2bow).

dictionary = gensim.corpora.Dictionary(articles_unfurled)

dictionary.save('plos_biology.dict')

dictionary = gensim.corpora.dictionary.Dictionary().load('plos_biology.dict')

I noticed that the word figure occurs rather frequently in these articles, so let us exclude this and any other words that appear in more than half of the articles in this data set (thanks to Radim for pointing this out to me).

dictionary.filter_extremes()

corpus = [dictionary.doc2bow(article) for article in articles_unfurled]

gensim.corpora.MmCorpus.serialize('plos_biology_corpus.mm', corpus)

corpus = gensim.corpora.MmCorpus('plos_biology_corpus.mm')

model = gensim.models.ldamodel.LdaModel(corpus, id2word=dictionary, update_every=1, chunksize=100, passes=2, num_topics=20)

model.save('plos_biology.lda_model')

model = gensim.models.ldamodel.LdaModel.load('plos_biology.lda_model')

And these are the twenty topics we find in 1,754 PLOS Biology articles:

for topic_i, topic in enumerate(model.print_topics(20)):
    print('topic # %d: %s\n' % (topic_i+1, topic))

Topics with Lemmatized Tokens

As we can notice, some of the tokens in the above topics are just singular and plural forms of the same word.

Let us see what topics we find after lemmatizing all of our tokens.

from nltk.stem import WordNetLemmatizer
wnl = WordNetLemmatizer()
articles_lemmatized = []
for article in articles_unfurled:
    articles_lemmatized.append([wnl.lemmatize(token) for token in article])

pickle.dump(articles_lemmatized, open('plos_biology_articles_lemmatized.list', 'w'))

dictionary_lemmatized = gensim.corpora.Dictionary(articles_lemmatized)

dictionary_lemmatized.save('plos_biology_lemmatized.dict')

dictionary_lemmatized.filter_extremes()

corpus_lemmatized = [dictionary_lemmatized.doc2bow(article) for article in articles_lemmatized]

gensim.corpora.MmCorpus.serialize('plos_biology_corpus_lemmatized.mm', corpus_lemmatized)

model_lemmatized = gensim.models.ldamodel.LdaModel(corpus_lemmatized, id2word=dictionary_lemmatized, update_every=1, chunksize=100, passes=2, num_topics=20)

for topic_i, topic in enumerate(model_lemmatized.print_topics(20)):
    print('topic # %d: %s\n' % (topic_i+1, topic))

Product ID	Timestamp	A	B	C	D	QA
…	…	…	…	…	…	…
123	2020-05-10 14:35:00	1.2	102.1	ZZ	YY	0.98
456	2020-05-10 16:20:00	1.1	104.3	ZZ	XX	0.96
789	2020-05-11 06:04:00	1.3	100.6	ZX	YY	0.83
012	2020-05-11 10:45:00	1.1	101.3	ZX	YZ	0.92
…	…	…	…	…	…	…

	age	job	marital	balance	duration	success
0	30	unemployed	married	1787	79	0
1	33	services	married	4789	220	0
2	35	management	single	1350	185	0
3	30	management	married	1476	199	0
4	59	blue-collar	married	0	226	0
...	...	...	...	...	...	...
4516	33	services	married	-333	329	0
4517	57	self-employed	married	-3313	153	0
4518	57	technician	married	295	151	0
4519	28	blue-collar	married	1137	129	0
4520	44	entrepreneur	single	1136	345	0

	success
duration_bucket
bucket 1	0.003932
bucket 2	0.031662
bucket 3	0.048193
bucket 4	0.095174
bucket 5	0.155378
bucket 6	0.358090

	AccPedal	AirIntakeTemperature	AmbientTemperature	BoostPressure	BrkVoltage	ENG_Trq_DMD	ENG_Trq_ZWR	ENG_Trq_m_ex	EngineSpeed_CAN	EngineTemperature	Engine_02_BZ	Engine_02_CHK	OilTemperature1	SCS_01_BZ	SCS_01_CHK	SCS_Cancel	SCS_Tip_Down	SCS_Tip_Set	SCS_Tip_Up	SteerAngle1	Trq_FrictionLoss	Trq_Indicated	VehicleSpeed	WheelSpeed_FL	WheelSpeed_FR	WheelSpeed_RL	WheelSpeed_RR	Yawrate1
0.000000	0.0	31.5	8.0	0.97	1.0	18.0	20.0	27.000000	809.500000	93.0	6.000000	168.000000	82.0	9.000000	27.000000	0.0	0.0	0.0	0.0	125.599998	29.0	27.000000	0.0	0.0	0.0	0.0	0.0	0.190000
0.050000	0.0	31.5	8.0	0.97	1.0	18.0	20.0	27.000000	809.500000	93.0	6.000000	168.000000	82.0	9.000000	27.000000	0.0	0.0	0.0	0.0	125.599998	29.0	27.000000	0.0	0.0	0.0	0.0	0.0	0.190000
0.100000	0.0	31.5	8.0	0.97	1.0	18.0	20.0	27.000000	810.215759	93.0	10.331579	163.663162	82.0	9.333000	26.000999	0.0	0.0	0.0	0.0	125.599998	29.0	27.000000	0.0	0.0	0.0	0.0	0.0	0.209900
0.150000	0.0	31.5	8.0	0.97	1.0	18.0	20.0	27.384237	807.657654	93.0	9.605911	165.674881	82.0	9.833000	24.500999	0.0	0.0	0.0	0.0	125.599998	29.0	27.384237	0.0	0.0	0.0	0.0	0.0	0.204887
0.200000	0.0	31.5	8.0	0.97	1.0	18.0	20.0	27.000000	805.500000	93.0	4.358586	172.641418	82.0	10.333000	24.333000	0.0	0.0	0.0	0.0	125.599998	28.0	27.000000	0.0	0.0	0.0	0.0	0.0	0.200000
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
7932.700195	0.0	25.5	9.5	0.98	0.0	20.0	19.0	29.000000	797.500000	96.0	8.174129	84.825874	96.0	10.493239	24.493240	0.0	0.0	0.0	0.0	14.100000	29.0	29.000000	0.0	0.0	0.0	0.0	0.0	0.130000
7932.750000	0.0	25.5	9.5	0.98	0.0	19.0	19.0	29.000000	800.500000	96.0	13.164251	145.835754	96.0	10.993991	24.993992	0.0	0.0	0.0	0.0	14.100000	29.0	29.000000	0.0	0.0	0.0	0.0	0.0	0.123609
7932.799805	0.0	25.5	9.5	0.98	0.0	19.0	19.0	29.000000	797.790588	96.0	2.177665	156.822342	96.0	11.494000	27.469999	0.0	0.0	0.0	0.0	14.100000	28.0	29.000000	0.0	0.0	0.0	0.0	0.0	0.178596
7932.850098	0.0	25.5	9.5	0.98	0.0	19.0	19.0	29.000000	796.000000	96.0	7.180095	153.739334	96.0	11.994000	29.969999	0.0	0.0	0.0	0.0	14.100000	28.0	29.000000	0.0	0.0	0.0	0.0	0.0	0.157268
7932.899902	0.0	25.5	9.5	0.98	0.0	19.0	19.0	29.000000	794.500000	96.0	12.155440	146.844559	96.0	12.494247	30.494247	0.0	0.0	0.0	0.0	14.100000	28.0	29.000000	0.0	0.0	0.0	0.0	0.0	0.140000

	AccPedal	AirIntakeTemperature	AmbientTemperature	BoostPressure	BrkVoltage	ENG_Trq_DMD	ENG_Trq_ZWR	ENG_Trq_m_ex	EngineSpeed_CAN	EngineTemperature	Engine_02_BZ	Engine_02_CHK	OilTemperature1	SCS_01_BZ	SCS_01_CHK	SCS_Cancel	SCS_Tip_Down	SCS_Tip_Restart	SCS_Tip_Set	SCS_Tip_Up	SteerAngle1	Trq_FrictionLoss	Trq_Indicated	VehicleSpeed	WheelSpeed_FL	WheelSpeed_FR	WheelSpeed_RL	WheelSpeed_RR	Yawrate1
0.000000	0.0	44.25	14.5	1.00	1.0	20.000000	21.0	30.000000	791.000000	96.0	6.000000	59.000000	94.0	10.000000	24.000000	0.0	0.0	0.0	0.0	0.0	4.800000	31.0	30.000000	0.0	0.0	0.0	0.0	0.0	0.140000
0.050000	0.0	44.25	14.5	1.00	1.0	21.000000	20.0	30.000000	791.000000	96.0	10.688680	102.688683	94.0	10.000000	24.000000	0.0	0.0	0.0	0.0	0.0	4.800000	31.0	30.000000	0.0	0.0	0.0	0.0	0.0	0.120000
0.100000	0.0	44.25	14.5	1.00	1.0	21.000000	20.0	30.000000	791.165039	96.0	4.381188	105.079208	94.0	10.110166	24.110165	0.0	0.0	0.0	0.0	0.0	4.800000	31.0	30.000000	0.0	0.0	0.0	0.0	0.0	0.148120
0.150000	0.0	44.25	14.5	1.00	1.0	21.000000	20.0	30.751268	787.500000	96.0	4.725888	104.725891	94.0	10.610916	24.610916	0.0	0.0	0.0	0.0	0.0	4.800000	31.0	30.751268	0.0	0.0	0.0	0.0	0.0	0.126384
0.200000	0.0	44.25	14.5	1.00	1.0	21.000000	20.0	31.000000	791.000000	96.0	9.733668	101.733665	94.0	11.111500	25.557501	0.0	0.0	0.0	0.0	0.0	4.800000	31.0	31.000000	0.0	0.0	0.0	0.0	0.0	0.135592
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
6807.450195	0.0	25.50	10.5	0.98	1.0	21.000000	21.0	31.000000	815.500000	94.5	10.693069	118.693069	95.0	5.992500	22.394501	0.0	0.0	0.0	0.0	0.0	37.400002	30.0	31.000000	0.0	0.0	0.0	0.0	0.0	0.130326
6807.500000	0.0	25.50	10.5	0.98	1.0	21.000000	21.0	30.682692	813.500000	94.5	5.192307	120.884613	95.0	0.100500	18.100500	0.0	0.0	0.0	0.0	0.0	37.400002	30.0	30.682692	0.0	0.0	0.0	0.0	0.0	0.150000
6807.549805	0.0	25.50	10.5	0.98	1.0	21.000000	21.0	31.000000	816.000000	94.5	4.676768	120.676765	95.0	0.600500	18.600500	0.0	0.0	0.0	0.0	0.0	37.476120	30.0	31.000000	0.0	0.0	0.0	0.0	0.0	0.140351
6807.600098	0.0	25.50	10.5	0.98	1.0	20.323383	21.0	30.000000	819.587524	94.5	9.676617	74.726372	95.0	1.100550	18.698349	0.0	0.0	0.0	0.0	0.0	37.476616	30.0	30.000000	0.0	0.0	0.0	0.0	0.0	0.140000
6807.649902	0.0	25.50	10.5	0.98	1.0	22.000000	21.0	30.722773	810.048096	94.5	14.693069	178.693069	95.0	1.600800	17.197599	0.0	0.0	0.0	0.0	0.0	37.500000	31.0	30.722773	0.0	0.0	0.0	0.0	0.0	0.129625

	month	spend	treatment
0	1	235	False
1	2	249	False
2	3	240	False
3	4	224	False
4	5	184	False
5	6	172	False
6	7	182	False
7	8	155	False
8	9	120	False
9	10	153	False
10	11	148	False
11	12	159	False

	user_id	month	high_spender	spend	signup_month	treatment
0	0	1	1	778	11	True
1	0	2	1	726	11	True
2	0	3	1	704	11	True
3	0	4	1	723	11	True
4	0	5	1	718	11	True
...	...	...	...	...	...	...
119995	9999	8	1	745	0	False
119996	9999	9	1	777	0	False
119997	9999	10	1	776	0	False
119998	9999	11	1	744	0	False
119999	9999	12	1	768	0	False

	age	job	marital	education	default	balance	housing	loan	contact	day	month	pdays	previous	poutcome	no_contacts
0	30	unemployed	married	primary	no	1787	no	no	cellular	19	oct	-1	0	unknown	1
1	33	services	married	secondary	no	4789	yes	yes	cellular	11	may	339	4	failure	1
2	35	management	single	tertiary	no	1350	yes	no	cellular	16	apr	330	1	failure	1
3	30	management	married	tertiary	no	1476	yes	yes	unknown	3	jun	-1	0	unknown	4
4	59	blue-collar	married	secondary	no	0	yes	no	unknown	5	may	-1	0	unknown	1

	age	balance
segment
high balance	44.542857	18361.771429
low balance	40.845577	543.930678
medium balance	42.911111	5202.864957

georg.io

The Machine Learning Toolbox

Communities

ML Collective

Kaggle

Concepts

Degenerate feedback loops

Further reading

Data distribution shift

Further reading

Frameworks

Pandas

When (not) to use it

Further reading

Causal inference for decision-making in growth hacking and upselling in Python

Introduction

Let's estimate the average treatment effect and make a decision!

References and further material ...

Install and load libraries

Randomized controlled trial / experimental data

Average treatment effect on post-signup spend for experimental data

The issue with randomized controlled trials and observational data

Identification / identifying the causal effect

Estimating the treatment effect

Causality modelling in Python for data scientists

Causality modelling in Python for data scientists

The need for causality modelling

How to do causality modelling

Use case: The impact of direct marketing on customer behavior

The direct marketing success data set

Trying to help our business with machine learning only

Uses for

The Causality-Driven Company

The data-driven company

Machine learning applications for the data-driven company

Smart manufacturing

Product quality prediction

Using our prediction model

Issue with our prediction model

The missing piece: causality

Variables and parameters are often interconnected

Interconnected variables are hard to disentangle

Causal inference toolset: Do-calculus

Becoming a causality-driven company

Decision intelligence from historical observations for optimal marketing resource use

Summary

Fetch the data

Load Python libraries

Prepare data

High-level model: more is better

Nuanced model: more isn't always better and there are always tradeoffs

Smart maintenance based on vehicle CAN bus data from scratch in Python

Summary

Fetch the data

Load libraries

Load vehicle data

Turn time series data into tables

Monitoring engine health

Engine 1 - healthy

Engine 2 - unhealthy

Anomaly detection algorithm: learn what observations to expect to recognize anomalies

Multiclass classification applied to a stream of documents in Python

Classifying Wikipedia Changes

Linkage discovery between scientific articles in Python and with graphs

PLOS Biology-Inspired PLOS Biology Articles

Topic discovery in scientific articles with Python

PLOS Biology Topics

Dictionary and Corpus Creation

Topics with Lemmatized Tokens