Diff and diff analysis

Diff-in-Diff (Difference-in-Differences)

Why?

Diff-in-diff is used to estimate the causal effect of a treatment when the treatment is not implemented as a randomized control trial (e.g. A/B testing).

Within Ad Market Dynamics we primarily use Diff-in-Diff to measure the impact of geo experiments.

When to use Diff-in-Diff Treatment is not randomly assigned (e.g. Geo experiment ✅  A/B testing ❌) Other things were happening while treatment was in effect You can't control for all the potential confounders Assumptions for Diff-in-Diff Trend in control group approximates what would have happened in the treatment group in the absence of the treatment i.e. if the treatment was not implemented, then the two groups would have experienced the same changes.  How to calculate Diff-in-Diff

An Example:

Diff-in-Diff for the treatment is calculated as follows: result is 15. 

Reference

  • Amazing Youtube video on Diff-in-Diff
  • Blog post with examples and case studies using Diff-in-Diff