Generating & Evaluating Synthetic Data for Predicting Drug Treatments

Comparing DP-CGAN and CTGAN for generating privacy-safe synthetic pharmacy data — presented at Vector Institute's Synthetic Data Bootcamp Demo Day

Presented at Vector Institute Demo Day — Synthetic Data Bootcamp, September 14, 2022. Authors: Jonathan Erez, Etienne Lin, John Chau, Ali Mahmood, Doaa Altarawy, Thas Yuwaraj, Tzvi Aviv — Loblaw Health Data Products and Services.

Overview

Pharmacy prescription data is rich with clinical signal — but its sensitivity under PIPEDA makes it difficult to share across teams, use for model development, or expose to external research partners. Synthetic data generation offers a path forward: if you can produce a dataset that is statistically indistinguishable from real patient records yet contains no real patient information, you can unlock the analytical value while managing privacy risk.

This project, developed in collaboration with the Vector Institute during their Synthetic Data Bootcamp, applied and evaluated two Conditional GAN (C-GAN) architectures on a real pharmacy dataset to assess their practical viability for predicting drug treatments.

Problem Setup

The task is a multiclass classification problem: given a patient’s features, predict their drug treatment class. The dataset consists of 8 features across 83,804 patients — 6 categorical and 2 continuous. The goal was to generate synthetic data that preserves the statistical properties of the real data closely enough to train accurate downstream models.

Two C-GAN Approaches

DP-CGAN (Differentially Private Conditional GAN) was based on Vector Institute’s reference implementation. We extended it to handle the multiclass classification setting, involving label encoding and one-hot encoding of categorical features. Differential privacy adds formal privacy guarantees at the cost of some fidelity.

CTGAN (Conditional Tabular GAN) comes from the Synthetic Data Vault (SDV) ecosystem. It is purpose-built for tabular data and addresses two key quality dimensions: likelihood fitness (does the synthetic data follow the same distributions?) and machine learning efficacy (does a model trained on synthetic data perform well on real data?).

Figure 1 — Age, Gender, and Drug Class Distributions (CTGAN)

Figure 1. Real vs. synthetic feature distributions generated by CTGAN. Age at initiation (AGE_AT_INIT) and patient gender (PTNT_GENDER) histograms show strong visual alignment between real (blue) and synthetic (orange) data. Drug class distributions by prescriber specialty are also preserved in the synthetic data.

The age distributions show CTGAN successfully capturing the shape of the real data — the characteristic peak around age 55–65 in the real pharmacy cohort is reproduced in the synthetic output. Gender proportions and specialty-level drug class breakdowns are similarly coherent.

Figure 2 — Cramer’s V Correlations Between Categorical Features (CTGAN)

Figure 2. Cramer's V correlation matrices for real data (left), CTGAN synthetic data (centre), and the difference (right). The correlation structure is largely preserved, with most pairwise differences below 0.15. The high correlation between feature_3 and feature_4 (0.73/0.75) is faithfully reproduced.

Cramer’s V measures the association between categorical feature pairs. A key concern with synthetic data is that learned correlations from the real data are preserved — particularly important when downstream models rely on inter-feature relationships. The difference matrix shows most deviations are small (< 0.15), and the strongest real correlation (feature_3 ↔ feature_4, 0.73) is reproduced at 0.75 in the synthetic data.

Figure 3 — Head-to-Head Evaluation: DP-CGAN vs. CTGAN

Figure 3. Evaluation results comparing DP-CGAN and CTGAN across five metrics. CTGAN wins on 3 of 5 metrics. DP-CGAN is faster (38 min vs. 1h 38 min) and achieves zero sample overlap, but CTGAN produces significantly better distributional fidelity (mean skew difference 17.5 vs. 86.5) and lower propensity scores (0.17 vs. 0.25).

Evaluation Metrics

Five metrics were used to assess synthetic data quality:

Feature distributions — visual inspection of histograms and bar charts comparing real and synthetic marginal distributions.

Correlation analysis — Cramer’s V for categorical features; ensures inter-feature relationships are preserved.

Subjective evaluation — domain experts presented with a random mix of real and synthetic records attempt to classify each as real or synthetic.

Distinguishability / Propensity score (pMSE) — a classifier is trained to distinguish real from synthetic records. Propensity scores closer to 0 indicate the two datasets are indistinguishable; closer to 1 indicates they are easily separable. CTGAN’s score of 0.17 vs. DP-CGAN’s 0.25 reflects meaningfully better fidelity.

Sample overlap score — what proportion of real patient records appear verbatim in the synthetic data? DP-CGAN achieves 0.00 (no overlap, stronger privacy), while CTGAN scores 0.01.

Results Summary

Metric DP-CGAN CTGAN
Mean correlation matrix difference 0.086 0.078
Propensity metric (pMSE) 0.25 0.17
Sample overlap score 0.00 0.01
Mean skew difference (continuous features) 86.5 17.5
Run time 38 mins 1h 38 mins

CTGAN wins on distributional fidelity (3 of 5 metrics) and is more user-friendly via the SDV Python package. DP-CGAN is faster and achieves strictly zero sample overlap — an advantage when formal differential privacy guarantees are required.

Conclusions

Both approaches generated coherent synthetic pharmacy datasets. CTGAN, via the SDV library, outperformed DP-CGAN on distributional fidelity metrics and is more accessible for tabular multiclass classification tasks. DP-CGAN’s differential privacy guarantees remain valuable in contexts requiring formal privacy proofs.

An important metric not yet measured: forecast accuracy of a model trained on synthetic data and tested on real data. This train-on-synthetic, test-on-real evaluation directly measures whether synthetic data can substitute for real data in model development pipelines — the ultimate practical test.

Future Directions

  • Epsilon and GAN hyperparameter tuning to improve the fidelity/privacy tradeoff in DP-CGAN
  • Train-on-synthetic, test-on-real classification benchmarks to measure downstream ML efficacy
  • Extending to multi-table and longitudinal pharmacy record structures