Skip to content

EDA Guidebook

🎯 Purpose

This guide outlines how to explore and prepare unlabeled data for clustering. It focuses on understanding structure, distribution, feature behavior, and conditions that influence model choices such as K-Means, DBSCAN, HDBSCAN, GMM, and hierarchical clustering.

🧠 1. Confirm Unsupervised Clustering Use Case

  • [ ] ✅ No target/label column (unsupervised setting)
  • [ ] ✅ Goal: segmentation, anomaly detection, or structure discovery
  • [ ] ✅ Interpretability vs flexibility prioritized (e.g., centroid profiles vs irregular shapes)

📊 2. Shape & Structure of the Dataset

🔹 Dimensionality Check

X.shape  # rows, features

✔️ Consider dimensionality reduction (PCA, UMAP) when d > 10

🔹 Missingness Summary

df.isnull().sum()

✔️ Ensure consistent imputation or deletion strategy

📦 3. Feature Distribution & Skew

🔹 Histograms & KDEs

sns.histplot(data=df, x='feature', kde=True)

✔️ Flag skewed variables for transformation ✔️ Right-skewed features may benefit from log scaling

🔹 Boxplots

sns.boxplot(data=df.select_dtypes(include='number'))

✔️ Identify univariate outliers

🔁 4. Scaling & Normalization Prep

🔹 Scaling Check

df.describe()  # compare mean, std, max across features

✔️ Normalize for KMeans, DBSCAN, GMM if using distance-based models ✔️ Tree-based distance methods (rare) may skip this step

🔹 Suggested Transformers

from sklearn.preprocessing import StandardScaler, MinMaxScaler

🧮 5. Correlation + Redundancy

🔹 Correlation Heatmap

sns.heatmap(df.corr(), cmap='coolwarm', annot=False)

✔️ Identify strongly correlated pairs ✔️ Consider PCA or feature pruning if redundancy > 0.85

🔍 6. Outlier Detection (Pre-Clustering)

🔹 Z-Score or IQR-Based Filtering

from scipy.stats import zscore
np.abs(zscore(df.select_dtypes(include='number'))) > 3

✔️ Remove/flag high outliers to avoid skewing cluster centers

🔹 Visual Outlier Inspection

  • Boxplot
  • PCA scatter with density shading

🌐 7. Dimensionality Reduction (for Visual Inspection)

🔹 PCA or UMAP for 2D Shape Review

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

✔️ Plot with color coding to visually inspect for clusters ✔️ Compare structure across scaled vs unscaled data

📋 Analyst EDA Checklist for Clustering

  • [ ] Dataset has no target label
  • [ ] Numeric and categorical features separated and reviewed
  • [ ] Distributions checked for skew, outliers, and scale variance
  • [ ] Features scaled using StandardScaler or MinMaxScaler
  • [ ] Correlation heatmap used to identify redundancy
  • [ ] PCA or UMAP previewed to inspect shape
  • [ ] Feature ranges aligned (avoid dominance by one feature)
  • [ ] Outlier strategy documented

💡 Final Tip

“Clustering finds what you feed it — scale, shape, and outliers are just as important as data volume.”

Use this before: K-Means, DBSCAN, GMM, HDBSCAN, Spectral, or Hierarchical clustering.