Chris Wen's Blog

Back groundI have a project that uses PostgreSQL as its database. The project has been running for a long time, and the production database has accumulated a large amount of historical data. The development database was originally synchronized with the production database. However, the existing data in the development environment is no longer needed. Therefore, I need to clear all existing data from the development database while preserving the database schema, reset identity columns, and mig...

SQL Server - Max Size ErrorIt is a common error in SQL Server when the database reaches its maximum size limit. This can happen due to various reasons, such as large data inserts, log file growth, or improper database configuration. It is easy to get this error. Here are some steps to resolve the issue: Check the current size of the database and its maximum size limit using the following query: 1234SELECT SUM(reserved_page_count) * 8.0 / 1024 AS ReservedMB, SUM(used_page_count) * 8.0 / 10...

Solution Architect - Associate 01S3What is S3?S3 (Simple Storage Service) is an object storage service that offers industry-leading durability, data availability, and security. S3 Standard: Frequently accessed; short-term/temporary data, typically stored for less than 30 days. S3 Intelligent-Tiering: Long-term storage; access frequency is unpredictable or you prefer not to classify data manually; automatic cost optimization is desired. S3 Standard-IA： Infrequently accessed, but data must be ...

Security - OWASP Top 10What is OWASP?The Open Web Application Security Project (OWASP) is a nonprofit organization focused on improving the security of software. It provides resources, tools, and best practices for developers to build secure applications. OWASP Top 10The OWASP Top 10 is a list of the most critical security risks to web applications. Broken Access Control: Users can access data or perform actions they are not authorized to use.Example: Changing /orders/123 to /orders/124 reve...

IntroductionJava Swing is a graphical user interface toolkit included in the Java Development Kit. It allows developers to build cross-platform desktop applications using standard Java. Swing provides many commonly used desktop components, including: Windows Buttons Text fields Drop-down lists Tables Dialog boxes Menu bars File selectors Although many modern applications are developed as web applications, Swing is still used in mature enterprise desktop systems, internal business applicatio...

What makes code maintainable?Maintainable code is code that is easy to understand, modify, test and extend. It should have clear naming, simple structure, small functions and consistent formatting. Good maintainable code follows SOLID principles, avoids unnecessary duplication, and separates responsibilities clearly. It should also include meaningful comments where needed, but the code itself should be readable without too many explanations. Automated tests, good error handling, and clear doc...

Certified Data Engineer Associate 20261. A data engineer has a DataFrame, salesDf, wilh a column named amount and wants to Permanently rename this column to total_amount for all future operations on salesDf In a separate query, temporarily show the column as gross_amount in the result, without changing the underlying schema Which set of code fragments meets the requirements? Operation 1: salesDf = salesDf.withColumnRenamed(“amount”, “total_amount”)Operation 2: salesDf.select(col(“*”), col(“a...

What is Databricks?Databricks is multi-cloud lakehouse platform based on Apache Spark. It provides a unified analytics platform that combines data engineering, data science, and machine learning capabilities. Databricks allows organizations to process and analyze large volumes of data efficiently, enabling them to derive insights and make data-driven decisions. Data Lake: data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. Y...

Digital Empowerment - Smart Future Predictive Maintenance: Anticipate equipment issues and reduce downtime and losses. IT/OT Integration: Break down information silos and enable seamless data connectivity. Intelligent Analytics: Data-driven decisions to optimize production and resources. Energy Efficiency: Monitor and optimize energy use to support green manufacturing. Emerging Trends Fewer resources Evoling technologies Higher production demands

Box-Cox and Yeo-Johnson TransformationsThe main motivations for transforming variables in data wrangling are to make data more suitable for analysis and modeling. Transformation often address specific challenges or requirements in the dataset, ensuring that it meets the assumptions of analytical methods or enhances interpretability and usability. In this post, we will discuss two common transformations: Box-Cox and Yeo-Johnson. Both transformations are used to transform skewed data to make it...