What is Databricks?
Databricks is multi-cloud lakehouse platform based on Apache Spark. It provides a unified analytics platform that combines data engineering, data science, and machine learning capabilities. Databricks allows organizations to process and analyze large volumes of data efficiently, enabling them to derive insights and make data-driven decisions.
Data Lake: data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.
Lakehouse; lakehouse is a new data management paradigm that combines the best features of data lakes and data warehouses. It provides the flexibility and scalability of a data lake while also offering the performance and reliability of a data warehouse. Lakehouses allow organizations to store and analyze all their data in a single platform, eliminating the need for separate systems and enabling faster insights.
Data warehouse: data warehouse is a centralized repository that allows you to store and manage structured data from various sources. It is designed for efficient querying and analysis, providing a high-performance environment for business intelligence and reporting. Data warehouses typically use a schema-on-write approach, where data is structured and organized before it is stored, making it easier to query and analyze.
Lakehouse Architecture
Control plane: Databricks control plane is responsible for managing the overall operation of the Databricks platform. It handles tasks such as user authentication, cluster management, job scheduling, and monitoring. The control plane ensures that the platform runs smoothly and efficiently, allowing users to focus on their data processing and analysis tasks.
- Web UI
- Cluster Management
- Workflow
- Notebooks
Data plane: Databricks data plane is responsible for executing the actual data processing and analysis tasks. It consists of clusters of compute resources that are used to run Spark jobs, perform data transformations, and execute machine learning algorithms. The data plane is designed to be scalable and flexible, allowing users to easily scale up or down their compute resources based on their workload requirements.
- Cluster VMs
- Storage(DBFS)
