Chris Wen's Blog

TerraformDocumentationterraform TofuDocumentationtofu

Question 1: Concurrent Web Crawler ImplementationDesign and implement a multi-threaded web crawler that efficiently crawls a specific domain and counts unique URLs Requirements: Handle circular references and invalid URLs implement rate limiting and exponential back off Process robots.txt compliance Support different URL schemes and redirects Optimize for memory usage with large datasets Overview12345678910111213141516171819[Seed URLs] | [Frontier in-memory queue] <--- [Persistent ov...

AWS Cloud FundamentalsAWS Cli Installationcli AWS Account Creation Root login: When you first create an AWS account, you’ll have a root user login linked to your email address. The root user has full control of the account! Guard this login carefully and use it only for tasks that require it. IAM users: AWS has a system called Identity and Access Management (IAM), which allows you to set up users and roles under your account that have exactly the permissions they need to do certain work. We’l...

Why Perform Performance Tuning? An online product without performance testing is like a ticking time bomb. You don’t know when it will break down, or what its limits are. Some performance problems accumulate slowly over time and eventually explode. Many more are caused by fluctuations in traffic. Performance tuning is a continuous process that helps you identify and address performance issues before they become critical, ensuring a smooth user experience and optimal resource utilization. Perf...

What is data transformationConditional Split TransformationThe conditional split transformation routes data rows to different streams based on matching conditions The conditional split transformation is similar to a CASE decision structure in a programming language. The transformation evaluates expressions, and based on the results, directs the data row to the specified stream Exists TransformationThe exists transformation is a row filtering transformation that checks whether your data exists...

What is ActivityCopy ActivityIn Azure Data Factory and Synapse pipelines, you can use the Copy activity to copy data among stores located on-premises and in the cloud. After you copy the data, you can use otheer activities to further transform and analyze it. Append Variable ActivityUse the Append Variable activity to add a value to an existing array variable. Delete ActivityDelete files or folders from on-premises storage stores or cloud storage store. This activity is used to clean up or ac...

Azure Data Factory OverviewAzure Data Factory (ADF) is a fully managed, cloud-based data integration service that allows you to create, schedule, and orchestrate data workflows.It is commonly used for ETL / ELT pipelines, enabling data movement and transformation across different data sources at scale. Resource GroupA Resource Group is a logical container in Azure that holds related resources for an Azure solution. In Azure Data Factory, the resource group typically contains: Azure Data Fact...

Flowable - FundationFlowable is a lightweight business process engine written in Java. It is designed to be fast, efficient, and easy to use, making it a popular choice for developers looking to implement workflow and business process management (BPM) solutions. A simple workflow might look like this:12345Start ↓Manager Approval ↓End 12345678910111213141516<?xml version="1.0" encoding="UTF-8"?><definitions xmlns="http://www.omg.org/spec/BPMN/20100524...

Installing Postgres using Docker Compose1234567891011121314151617version: "3.9"services: postgres: image: postgres:13.22-trixie container_name: postgres_13 restart: always environment: POSTGRES_USER: root POSTGRES_PASSWORD: root POSTGRES_DB: mydb ports: - "5432:5432" volumes: - ./data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U root"]

Installing Mysql using Docker Compose12345678910111213141516version: '3'services: mysql: image: mysql:8.0.18 container_name: mysql8 environment: - MYSQL_ROOT_PASSWORD=root# - TZ=Asia/Shanghai volumes: - ./log:/var/log/mysql - ./data:/var/lib/mysql - ./conf/conf.d:/etc/mysql/conf.d # - /etc/localtime:/etc/localtime:ro ports: - 3306:3306 restart: always