Imbalanced Data
Imbalanced DataImbalanced data research The editorial on Learning from Imbalanced Data highlights the central challenge that in many real-world applications—such as fraud detection, medical diagnosis, and anomaly detection—the important class is often rare, causing standard machine-learning algorithms to bias heavily toward the majority class. It summarizes three major research directions: data-level methods (oversampling, undersampling, synthetic sample generation), algorithm-level methods (...
Outlier
Outliers123456import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snspd.set_option("display.notebook_repr_html", False) # disable "rich" outputplt.style.use("seaborn") Unidimensional Data1234567x = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/blobs2.txt")plt.subplot(121)sns.boxplot(data=x, orient="h")plt.subplot(122)sns.histplot(x, binwidth=1)plt.show() M...
Exploratory Data Analysis
Exploratory Data AnalysisEDA%20is,step%20in%20any%20data%20analysis.) Exploratory Data Analysis(EDA) is the process of looking at data before doing any detailed analysis. Its goal is to understand what the data looks like, find mistakes or unusual values, and discover simple patterns. In EDA, we check how individual variables are distributed, see whether two variables seem related, and identify outliers that may need attention. This is often done using simple charts and basic statistics. EDA ...
Continuous Probability Distributions
Continuous Probability Distributions1234import numpy as npimport matplotlib.pyplot as pltimport seaborn as snsplt.style.use("seaborn") 123456heights = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/nhanes_adult_female_height_2020.txt")sns.histplot(heights, stat="density", kde=True)plt.show()import scipy.stats Normal DistributionNormal Distribution12345678μ = np.mean(heights) # an estimator of expected valueσ =...
Multivariate Categorical and Relational Data
Multivariate Categorical and Relational Data12345import numpy as npmarathon = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/37_pzu_warsaw_marathon_3groups_top1000.txt", delimiter=",", dtype=str)marathon[:6, :] # preview Two-Way Contingency Tables123np.unique(marathon[:, 0])np.unique(marathon[:, 1]) 123456import scipy.statsl, v = scipy.stats.contingency.crosstab(marathon[:, 0], marathon[:, 1])l, vimport marekmarek.pr...
Visualising Multidimensional Data and Measuring Correlation
Visualising Multidimensional Data and Measuring Correlation123456789import numpy as npimport pandas as pdbody = pd.read_csv("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/nhanes_adult_female_bmx_2020.csv", comment="#")body = body.to_numpy() # data frames will be covered laterbody.shapebody[:6, :] # 6 first rows, all columns Scatterplots2D Data1234567891011import matplotlib.pyplot as pltimport seaborn as snsplt.style.use("...
Handling Categorical Data
Handling Categorical DataRepresenting Categorical DataTwo common ways to represent a categorical variable with k distinct levels is by storing it as: a vector of strings, a vector of integers between 0 (inclusive) and k (exclusive). 12345678import numpy as npcountries = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/37_pzu_warsaw_marathon_country.txt", dtype="str")x = countries[:16]xnp.unique(x) Encoding and Decodin...
Inspecting the Distribution of Numberic Data
Inspecting the Distribution of Numberic Data12345import numpy as npheights = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/nhanes_adult_female_height_2020.txt")np.random.choice(heights, 24, replace=False) Histograms1234567import matplotlib.pyplot as pltimport seaborn as snsplt.style.use("seaborn")sns.__version__ # FYIsns.histplot(heights, bins=11)plt.show() 1234income = np.loadtxt("https://raw.githubusercontent.co...
Descriptive Statistic for Continuous Data
Descriptive Statistic fopr Continuous DataHistograms are based on binned data and hence provide us with snapshots of how much probability mass is allocated in diferent parts of the data domain. 1234567import numpy as npincome = np.loadtxt("https://raw.githubusercontent.com/gagolews/" + "teaching-data/master/marek/uk_income_simulated_2020.txt")b = [0, 10000, 20000, 30000, 40000, 50000, 60000, 80000, np.inf] # bin boundsc = np.histogram(income, bins=b)[0] # countsfor i ...
Python - Basic Usage of Data Analysis
Basic Usage of Data AnalysisGetting Started with JupyterLabJupyterLab is a web-based development environment supporting numerous programming languages, including, of course, Python. jupyterlab Scalar Types in PythonBasic Operations on Data Framesdoc pre-requisite123import numpy as npimport pandas as pdpd.set_option("display.notebook_repr_html", False) # disable "rich" output Aggregating1234567891011121314151617181920np.random.seed(123)d = pd.DataFrame(dict( u = np.roun...
