Chris Wen's Blog

IntroductionJava Swing is a graphical user interface toolkit included in the Java Development Kit. It allows developers to build cross-platform desktop applications using standard Java. Swing provides many commonly used desktop components, including: Windows Buttons Text fields Drop-down lists Tables Dialog boxes Menu bars File selectors Although many modern applications are developed as web applications, Swing is still used in mature enterprise desktop systems, internal business applicatio...

What makes code maintainable?Maintainable code is code that is easy to understand, modify, test and extend. It should have clear naming, simple structure, small functions and consistent formatting. Good maintainable code follows SOLID principles, avoids unnecessary duplication, and separates responsibilities clearly. It should also include meaningful comments where needed, but the code itself should be readable without too many explanations. Automated tests, good error handling, and clear doc...

Certified Data Engineer Associate 20261. A data engineer has a DataFrame, salesDf, wilh a column named amount and wants to Permanently rename this column to total_amount for all future operations on salesDf In a separate query, temporarily show the column as gross_amount in the result, without changing the underlying schema Which set of code fragments meets the requirements? Operation 1: salesDf = salesDf.withColumnRenamed(“amount”, “total_amount”)Operation 2: salesDf.select(col(“*”), col(“a...

What is Databricks?Databricks is multi-cloud lakehouse platform based on Apache Spark. It provides a unified analytics platform that combines data engineering, data science, and machine learning capabilities. Databricks allows organizations to process and analyze large volumes of data efficiently, enabling them to derive insights and make data-driven decisions. Data Lake: data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. Y...

Digital Empowerment - Smart Future Predictive Maintenance: Anticipate equipment issues and reduce downtime and losses. IT/OT Integration: Break down information silos and enable seamless data connectivity. Intelligent Analytics: Data-driven decisions to optimize production and resources. Energy Efficiency: Monitor and optimize energy use to support green manufacturing. Emerging Trends Fewer resources Evoling technologies Higher production demands

Box-Cox and Yeo-Johnson TransformationsThe main motivations for transforming variables in data wrangling are to make data more suitable for analysis and modeling. Transformation often address specific challenges or requirements in the dataset, ensuring that it meets the assumptions of analytical methods or enhances interpretability and usability. In this post, we will discuss two common transformations: Box-Cox and Yeo-Johnson. Both transformations are used to transform skewed data to make it...

Feature SelectionAn Introduction to Variable and Feature Selection This paper provides an introduction to the topic of variable and feature selection, which has become increasingly important in fields with high-dimensional datasets such as text processing, gene expression analysis, and combinatorial chemistry. The authors discuss the objectives of variable selection, which include improving prediction performance, reducing measurement and storage requirements, and gaining a better understandi...

General Data Protection RegulationGDPRThe General Data Protection Regulation(GDPR) is a comprehensive data protection law enacted by the European Union to enhance individuals’ control over their personal data to harmonise data protection rules across EU member states. GDPR was introduced in response to rapid technological developments and the increasing scale of personal data processing in the digital economy. GDPR emphasises the protection of individuals’ fundamental rights and freedoms, par...

Dimensionality Reduction and PCADimensionality Reduction: A Comparative Review Dimensionality reduction is not just a technical trick. It’s a way of seeing data more clearly, almost like finding the right angle to view a complex sculpture. By reducing dimensions, we reveal the “shape” of the data—clusters, patterns, and relationships that would otherwise be hidden. Principal component analysis (PCA) is a fancy name for the whole process of reflecting upon what happens along the projections on...

Data MatchingData Matching After reading `Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection, I gained a clearer understanding of the challenges involved in working with real-world data. The book highlights that data is often incomplete, inconsistent, and noisy, and that data matching aims to identity records referring to the same real-world entity under these imperfect conditions. Rather than focusing on a single algorithm, the book presents...