Chris Wen's Blog

Box-Cox and Yeo-Johnson TransformationsThe main motivations for transforming variables in data wrangling are to make data more suitable for analysis and modeling. Transformation often address specific challenges or requirements in the dataset, ensuring that it meets the assumptions of analytical methods or enhances interpretability and usability. In this post, we will discuss two common transformations: Box-Cox and Yeo-Johnson. Both transformations are used to transform skewed data to make it...

Feature SelectionAn Introduction to Variable and Feature Selection This paper provides an introduction to the topic of variable and feature selection, which has become increasingly important in fields with high-dimensional datasets such as text processing, gene expression analysis, and combinatorial chemistry. The authors discuss the objectives of variable selection, which include improving prediction performance, reducing measurement and storage requirements, and gaining a better understandi...

General Data Protection RegulationGDPRThe General Data Protection Regulation(GDPR) is a comprehensive data protection law enacted by the European Union to enhance individuals’ control over their personal data to harmonise data protection rules across EU member states. GDPR was introduced in response to rapid technological developments and the increasing scale of personal data processing in the digital economy. GDPR emphasises the protection of individuals’ fundamental rights and freedoms, par...

Dimensionality Reduction and PCADimensionality Reduction: A Comparative Review Dimensionality reduction is not just a technical trick. It’s a way of seeing data more clearly, almost like finding the right angle to view a complex sculpture. By reducing dimensions, we reveal the “shape” of the data—clusters, patterns, and relationships that would otherwise be hidden. Principal component analysis (PCA) is a fancy name for the whole process of reflecting upon what happens along the projections on...

Data MatchingData Matching After reading `Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection, I gained a clearer understanding of the challenges involved in working with real-world data. The book highlights that data is often incomplete, inconsistent, and noisy, and that data matching aims to identity records referring to the same real-world entity under these imperfect conditions. Rather than focusing on a single algorithm, the book presents...

Imbalanced DataImbalanced data research The editorial on Learning from Imbalanced Data highlights the central challenge that in many real-world applications—such as fraud detection, medical diagnosis, and anomaly detection—the important class is often rare, causing standard machine-learning algorithms to bias heavily toward the majority class. It summarizes three major research directions: data-level methods (oversampling, undersampling, synthetic sample generation), algorithm-level methods (...

Outliers123456import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snspd.set_option("display.notebook_repr_html", False) # disable "rich" outputplt.style.use("seaborn") Unidimensional Data1234567x = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/blobs2.txt")plt.subplot(121)sns.boxplot(data=x, orient="h")plt.subplot(122)sns.histplot(x, binwidth=1)plt.show() M...

Exploratory Data AnalysisEDA%20is,step%20in%20any%20data%20analysis.) Exploratory Data Analysis(EDA) is the process of looking at data before doing any detailed analysis. Its goal is to understand what the data looks like, find mistakes or unusual values, and discover simple patterns. In EDA, we check how individual variables are distributed, see whether two variables seem related, and identify outliers that may need attention. This is often done using simple charts and basic statistics. EDA ...

Continuous Probability Distributions1234import numpy as npimport matplotlib.pyplot as pltimport seaborn as snsplt.style.use("seaborn") 123456heights = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/nhanes_adult_female_height_2020.txt")sns.histplot(heights, stat="density", kde=True)plt.show()import scipy.stats Normal DistributionNormal Distribution12345678μ = np.mean(heights) # an estimator of expected valueσ =...

Multivariate Categorical and Relational Data12345import numpy as npmarathon = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/37_pzu_warsaw_marathon_3groups_top1000.txt", delimiter=",", dtype=str)marathon[:6, :] # preview Two-Way Contingency Tables123np.unique(marathon[:, 0])np.unique(marathon[:, 1]) 123456import scipy.statsl, v = scipy.stats.contingency.crosstab(marathon[:, 0], marathon[:, 1])l, vimport marekmarek.pr...