Chris Wen's Blog

System Design - SeminarWhat is System Design?System design is the process of defining the architecture, components, modules, interfaces, and data for a system to satisfy specified requirements. It involves making high-level decisions about the structure and behavior of a system, including how it will interact with users, other systems, and its environment.

Certified Data Engineer Associate 02101. A data engineer has realized that the data files associated with a Delta table are incredibly small. They want to compact the small files to form larger files to improve performance. Which keyword can be used to compact the small files?Answer: OPTIMIZE OPTIMIZE table_name: Compacts small files into larger files to improve performance.VACUUM table_name: Removes old files that are no longer needed after compaction. 102. A data engineer wants to create ...

Certified Data Engineer Associate 011. A data organization leader is upset about the data analysis team’s reports being different from the data engineering team’s reports. The leader believes the siloed nature of their organization’s data engineering and data analysis architectures is to blame. Which of the following describes how a data lakehouse could alleviate this issue?Answer: Both teams would use the same source of truth of their work. Databricks Lakehouse enables using data as the sin...

Overview of Machine LearningWhat is Machine Learning?Machine Learning (ML) is a subset of artificial intelligence (AI) that enables computers to learn from data and improve their performance over time without being explicitly programmed. ML algorithms build mathematical models based on sample data, known as “training data,” to make predictions or decisions without being explicitly programmed to perform the task. Artificial intelligence is a field of computer science that aims to create machi...

ProbabilityProbability is a measure of how likely an event is to occur.The probability of an event ( A ) is denoted as: P(A)Probability Calculations Event Formula Complement of A P(A^c) = 1 - P(A) Joint Probability (A and B) P(A \cap B) Union Probability (A or B) P(A \cup B) = P(A) + P(B) - P(A \cap B) Conditional Probability P(A \mid B) = \frac{P(A \cap B)}{P(B)} ExampleA bag contains 10 balls: 6 red and 4 blue. Two balls are drawn randomly without replacement. Let: Event ...

Linear AlgebraScalars and Vectors Scalar: A single number representing magnitude only. Vector: An ordered array of scalars representing both magnitude and direction. Row Vector: $\mathbf{x} = [x_1, x_2, \dots, x_n]$ Column Vector: $\mathbf{x} = \begin{bmatrix} x_1 \ x_2 \ \vdots \ x_n \end{bmatrix}$ Vector Operations Transpose: Converts a column vector into a row vector.\mathbf{x}^T = [x_1, x_2, \dots, x_n] Addition: Element-wise addition of two vectors.\mathbf{x} + \mathbf{y} = [x_1+y_1, ...

DerivativeThe derivative is a fundamental concept in calculus. The derivative of a function at a specific point represents the rate of change of the function near that point (i.e., the slope of the tangent line to the function’s graph at that point). The essence of a derivative is to provide a local linear approximation of a function using the concept of limits. When the independent variable of function $f(x)$ produces an increment $\Delta x$ at point $x_0$, if the limit of the ratio between...

Why can a Java request hang when CPU and network look normal?When CPU and network are normal, the request is usually waiting for a resource or blocked by another operation, rather than actively executing. Here are some common causes: Slow or blocked database operations Thread deadlock or lock contention Web server thread pool exhaustion Downstream service calls without timeouts CompletableFuture or asynchronous-task starvation Transaction or distributed lock is never released Garbage collect...

1. What is Java, and what are the features of Java?Java is a high-level program language and is platform-independent.Some of the key features of Java include: Object-Oriented: Java is an object-oriented programming language, which means it allows developers to create modular programs and reusable code. Platform Independence: Java code can run on any platform that has a Java Virtual Machine (JVM), making it highly portable. Robust: Java has strong memory management and exception handling feat...

Interview QuestionsCore Java (OOPs, Collections, Concurrency Basics)Explain difference between HashMap, HashTable, and ConcurrentHashMap. Feature HashMap Hashtable ConcurrentHashMap Thread Safety Not thread-safe Thread-safe (synchronized methods) Thread-safe (lock segmentation / fine-grained locking) Null Keys/Values Allows one null key and multiple null values Does not allow any null keys/values Does not allow null keys/values Performance Faster (no synchronization overhead) Slowe...