Certified Data Engineer Associate 2026

1. A data engineer has a DataFrame, salesDf, wilh a column named amount and wants to

Permanently rename this column to total_amount for all future operations on salesDf
In a separate query, temporarily show the column as gross_amount in the result, without changing the underlying schema

Which set of code fragments meets the requirements?

Operation 1: salesDf = salesDf.withColumnRenamed(“amount”, “total_amount”)
Operation 2: salesDf.select(col(“*”), col(“amount”).alias(“gross_amount”))

2. A data engineer builds a Lakeflow Job that ingests JSON logs from cloud storage. New files arrive at unpredictable times, and the team wants to automatically run the job as soon as new files land so the data is processed immediately. What trigger configuration should the data engineer use for this job?

File arrival trigger
In Databricks Lakeflow Jobs, a File Arrival trigger is designed to monitor a specified cloud storage path. This can be a Unity Catalog volume or an external location such as Amazon S3, Azure Storage, or Google Cloud Storage. The trigger starts a job run when new files arrive in the monitored location. This feature is useful when scheduled jobs become inefficient due to irregular data arrival patterns.

Continuous trigger: Keeps a job running indefinitely by launching a new run immediately after the previous one finishes or fails. It is optimized for continuous stream processing rather than reacting dynamically to newly dropped batch files.
Scheduled trigger: Runs jobs at specific, fixed intervals (e.g., every hour or daily at 12:00 AM). Because the logs arrive at completely unpredictable times, a scheduled trigger would cause latency or waste compute resources processing empty paths.
Table update trigger: Listens for changes/commits made to existing Delta or Unity Catalog tables, rather than checking a raw cloud storage landing zone for new unparsed JSON log files.

3. A data engineering team builds a Lakeflow Job with an ingestion task followed by two possible execution paths: one for streaming processing and one for batch processing. They want to select the appropriate path dynamically based on a parameter processing_mode.

Use an if/else conditional task to route execution based on the processing_mode parameter.
Databricks Lakeflow Jobs explicitly provide an If/else condition task type designed to introduce boolean branching logic into your Directed Acyclic Graph (DAG) workflows.
By setting up an If/else task, you can evaluate runtime parameters (such as processing_mode) directly at the orchestration level. Based on whether the condition evaluates to true or false, Lakeflow will dynamically route the execution down the appropriate downstream path (streaming or batch), leaving the irrelevant branch untouched and saving computational resources.

4. A data engineer is trying to use Delta time travel feature to rollback a table to a previous version, but the data engineer received an error that the data files are no longer present. Which of the following commands was run on the table that caused deleting the data files?

Vacuum command
The Delta Lake vacuum command is used to clean up old data files that are no longer referenced by the Delta table. When a vacuum operation is performed, it permanently deletes the data files that are older than the retention period (default is 7 days). If a data engineer attempts to use Delta time travel to access a previous version of the table after the vacuum command has been executed, they will encounter an error indicating that the data files are no longer present, as those files have been permanently removed.

5. A data engineer is working with a PySpark DataFrame df that contains online store visits. Each row represents a customer viewing a product page, with several columns including: user_id, product_id, and timestamp. They want to ensure uniqueness at the product-view level, meaning each user should only have one record per product. Which of the following PySpark code correctly implements this logic?

1	df.dropDuplicates(["user_id", "product_id"])

The goal is to ensure uniqueness at the product-view level per user, meaning a single user should only have one recorded view for a specific product.

In PySpark, the dropDuplicates() (or its alias distinct()) method removes duplicate rows from a DataFrame. When you pass a list of column names to dropDuplicates([cols]), PySpark considers rows to be duplicates if they have identical values only in those specified columns. It then keeps the first occurrence it encounters and drops the rest.

6. “A feature that illustrates the relationship between different data assets including tables, queries, notebooks, and dashboards, enabling users to trace the origin and flow of data across the entire lakehouse platform.” Which of the following is being described in the above statement?

Unity Catalog Data Lineage

The description accurately defines the Data Lineage feature within Unity Catalog. This functionality is designed to provide transparency into data flows, allowing users to trace how datasets are created, modified, and consumed throughout the Databricks workspace. It is an essential tool for data governance, auditing, and ensuring the reliability of analytics workflows.

7. According to the Databricks Lakehouse architecture, which of the following is located in the customer’s cloud account?

Classic compute virtual machines

When the customer sets up a classic compute, the cluster virtual machines are deployed in the data plane in the customer’s cloud account.

8. A data engineer is exploring a large dataset in Databricks and executes df.summary() to better understand the data.

A DataFrame containing statistical metrics, including count, mean, standard deviation, min, max, and approximate quartile values for each column.

The df.summary() command is used for exploratory data analysis. It computes specified statistics for all numerical and string columns in a DataFrame.

When you run df.summary() without any specific arguments, it automatically calculates the following metrics for each applicable column:

count (number of non-null values)
mean
stddev (standard deviation)
min
25% (approximate 1st quartile)
50% (median / approximate 2nd quartile)
75% (approximate 3rd quartile)
max

The output itself is returned as a standard Spark DataFrame, which you can then view using display(df.summary()) in a Databricks notebook.

9. A data engineering team has the following Declarative Automation Bundle (DAB) configuration:

variables:
 schema_name:
   default: demo_schema
 
resources:
 jobs:
   sales_job:
     name: sales_job
     tasks:
       - task_key: load_data
         notebook_task:
           notebook_path: ../src/load_data.py
           base_parameters:
             schema_name: ${var.schema_name}

A data engineer deploys the bundle using:

1	databricks bundle deploy -t dev --var="schema_name=finance_schema"

Later, the data engineer runs the job using:

1	databricks bundle run -t dev sales_job --var="schema_name=marketing_schema"

What value will be passed to the notebook parameter schema_name during the job run?

finance_schema

Deployment-Time Interpretation: Bundle variables (${var.schema_name}) are deployment-time concepts in Declarative Automation Bundles (DABs). When you execute databricks bundle deploy -t dev --var="schema_name=finance_schema", the CLI completely evaluates all the custom variables and substitutes them directly into the deployed job definition configuration in the workspace.
Therefore, the deployed job definition is explicitly written with:
1
2
base_parameters:
schema_name: finance_schema
Run-Time Evaluation: When you later execute databricks bundle run, it tells Databricks to trigger the already-deployed job configuration. The --var flag is explicitly a deployment-time modifier for variable evaluation; it cannot dynamically manipulate standard bundle variable references at runtime during a bundle run.
Because the schema_name parameter was already hardcoded as finance_schema in the workspace upon deployment, that is the value passed to the notebook.

10. A data engineering team is developing a complex ETL pipeline on Databricks. They want to ensure that their workflow configurations are version-controlled and can be deployed reliably across staging and production environments. What is the most appropriate solution to achieve this task?

Store the source code in Git folders, and deploy jobs using Databricks Asset Bundles (DAB)

Databricks Asset Bundles allow teams to define jobs and workflows declaratively in a YAML file, promoting consistency, version control, and automation. These configurations can be stored and tracked in GitHub, making deployments reproducible and reliable. Other methods rely on manual processes or ad hoc tooling and lack the benefits of structured DevOps practices.

11. A data analyst at a retail company is responsible for generating daily reports on sales performance across multiple regions and product categories. The company ingests transaction data continuously from its online stores using Lakeflow Declarative Pipelines (formerly Delta Live Tables). The analyst needs to create a relational object that can efficiently precompute business-level aggregations, such as total revenue, average order value, and units sold per category, so that downstream reporting and dashboards can access the data quickly without recalculating it every time. Which of the following objects is most suitable for this use case?

Materialized View

The most suitable object for this use case is a materialized view because it allows the data analyst to precompute and store business-level aggregations, such as total revenue, average order value, and units sold per category, so that downstream reports and dashboards can access the results quickly without recalculating them every time, unlike a temporary or standard view, which either exist only for the session or require repeated recomputation, and unlike a streaming table, which is designed for processing raw, real-time event streams rather than pre-aggregated summaries.

12. Which of the following best describes the correct format of a Databricks Asset Bundle configuration file?

A YAML file named databricks.yml that defines the bundle’s structure, including targets, resources, and configurations.

Databricks Asset Bundles use a structured YAML configuration file named databricks.yml as the primary entry point. This file defines key aspects of the bundle, such as:

Targets: Represent different environments (e.g., dev, staging, prod).
Resources: Define objects like jobs, notebooks, pipelines, and clusters.
Configurations: Include settings such as workspace paths, environment variables, and parameter values.

The format is designed to be declarative and human-readable, facilitating clear version control and automation within CI/CD workflows. Unlike JSON or TOML, YAML is preferred for its readability and native support for nested structures, making it ideal for complex infrastructure and deployment definitions.

bundle:
    name: my_bundle
 
targets:
    dev:
        default: true
 
    prod:
        workspace:
            host: https://<production-workspace-url>
 
resources:
    jobs:
        my-job

13. Which of the following statements best describes Auto Loader?

Auto loader monitors a source location, in which files accumulate, to identify and ingest only new arriving files with each command run. While the files that have already been ingested in previous runs are skipped.

Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage.

14. A data engineering team is ingesting 100,000 new files of small JSON files per day into a Delta table from Azure Data Lake Storage. The team requires low-latency incremental ingestion and wants to minimize storage API costs caused by frequent directory scans. Which ingestion configuration should the team prioritize?

Use Auto Loader with cloudFiles.useNotifications = true to enable event-based file detection

cloudFiles.useNotifications = true is an option in Databricks Auto Loader that switches the ingestion mechanism from directory listing to an event-driven notification system.

Instead of repeatedly scanning an ADLS folder to detect new files, Auto Loader uses Azure Event Grid notifications to learn when new files arrive.
So the flow becomes:

A new file lands in Azure Data Lake Storage (ADLS)
ADLS emits an event via Event Grid
Databricks Auto Loader receives the event
Only that file is picked up for processing

Without notifications, Auto Loader uses:

Directory listing mode → periodically scans folders to find new files

That works, but at very large scale (like 100,000 files/day), listing becomes:

slower
more expensive (API calls)
less efficient

Benefits of useNotifications = true

Faster ingestion latency (near real-time detection)
Lower storage/listing costs (fewer ADLS list operations)
Better scalability for very high file volumes
Still supports incremental processing + checkpointing
Works well with schema inference and evolution
Maintains Unity Catalog lineage when writing to governed tables

It’s ideal when:

You have very high file arrival rates
Your storage supports Event Grid integration (like ADLS Gen2)
You want minimal overhead and near real-time ingestion

Simple mental model

Without notifications → “Databricks keeps checking the folder”
With notifications → “Storage tells Databricks when something new arrives”

15. A data engineer uses the following SQL query:

1	GRANT MODIFY ON TABLE employees TO hr_team

Which of the following describes the ability given by the MODIFY privilege ?

It gives the ability to add, update, or delete data within the table

The MODIFY privilege grants a user the ability to add, update, or delete data from an object. It is typically assigned to users who need to make changes to existing records or insert new information into the object.

16. A data engineering team is building a notebook and needs fast iteration and debugging during development. Which compute should they use in this case?

Interactive compute

Interactive compute resources are specifically designed for analyzing data collaboratively using interactive notebooks. They allow data engineers and data scientists to manually start the compute, write code cell-by-cell, run it immediately, and see the results in real time. Features like the Python Notebook Interactive Debugger, breakpoint insertion, and variable inspection are uniquely built into this environment, making it the ideal choice for rapid development and troubleshooting.

Serverless SQL warehouse: These are highly optimized compute resources specifically built to execute SQL queries, power BI dashboards, and run analytics workloads. They are not intended for broad data engineering notebook tasks (such as writing complex Python, Scala, or PySpark ETL pipelines) or general code debugging.

Serverless Job compute: Job compute (automated compute) is designed for running automated, production-level scheduled tasks and pipelines via Databricks Workflows. When a job runs, Databricks creates the compute resource, runs the task, and terminates it immediately. Because it does not support active, live iteration or cell-by-cell execution with an attached debugger, it is poorly suited for the active development phase.

Instance pools: An instance pool is a set of idle, ready-to-use virtual machine instances that reduce cluster start and auto-scaling times. While they can be used to back interactive clusters, an instance pool itself is an infrastructure configuration layer—not a type of compute workload meant for running and debugging a notebook.

17. A data engineer has made changes to a notebook inside a Databricks Git folder on a feature branch. The engineer now wants to merge these changes into the main branch. What is the correct workflow for performing this operation?

Commit and push changes to the remote feature branch, then use the GitHub UI or GitHub CLI to create a pull request (PR).

Databricks Git folders allow you to perform local Git operations such as creating branches, staging files, committing, pushing, pulling, and rebasing directly inside your workspace.

However, the Databricks UI does not natively support creating pull requests (PRs) or managing code reviews. To safely merge a feature branch into a protected branch like main, the standard collaborative Git workflow dictates that you push your commits from Databricks to the remote repository (e.g., GitHub, GitLab, Bitbucket). Once the changes are available on the remote, you must use your Git provider’s interface (such as the GitHub UI or GitHub CLI) to open, review, and ultimately merge the PR.

18. In the Medallion Architecture, which of the following statements best describes the Silver layer tables?

They provide a more refined view of raw data, where it’s filtered, cleaned, and enriched.

Silver tables provide a more refined view of the raw data. For example, data can be cleaned and filtered at this level. And we can also join fields from various bronze tables to enrich our silver records.

19. A data analyst works with two PySpark DataFrames: products_df and reviews_df, both containing a product_id column. They need a DataFrame that includes all products, even if they have no reviews, and enriches it with review data where available. Which PySpark code achieves this?

joined_df = products_df.join(reviews_df, “product_id”, “left”)

In PySpark data manipulation, a left join (left or left_outer) keeps all rows from the left DataFrame (products_df) regardless of whether there is a match in the right DataFrame (reviews_df).

The Left DataFrame (products_df): Represents the primary entity. Every single record is preserved.
The Right DataFrame (reviews_df): If a matching product_id is found, the review details are appended to that row. If no match exists, the missing columns are automatically populated with null.

inner: Only retains rows where there is a match in both DataFrames (this would mistakenly drop products that haven’t received reviews yet).
full / full_outer: Retains all rows from both DataFrames, which would introduce extra rows for reviews that don’t map to a valid product.
cross: Computes the Cartesian product of both DataFrames, pairing every single product with every single review, which is computationally expensive and logically incorrect for this use case.

20. Which of the following statements is Not true about Delta Lake ?

Delta Lake builds upon standard data formats: Parquet + XML

Delta Lake provides ACID transaction guarantees
Delta Lake provides scalable data and metadata handling
Delta Lake provides audit history and time travel

It is not true that Delta Lake builds upon XML format. It builds upon Parquet and JSON formats

21. A data engineer needs to schedule a daily job to read data from an on-premises PostgreSQL database hosted inside a corporate data center. They must choose the appropriate compute option to ensure reliable connectivity to the on-prem system. Which compute type should the data engineer use?

Classic job compute

Classic compute resources (such as classic job clusters) are deployed directly within your company’s own cloud account virtual network (VNet on Azure, VPC on AWS/GCP), a setup known as VNet injection. Because the compute instances live entirely inside your cloud network, they can seamlessly leverage established corporate routing infrastructure—such as a Site-to-Site VPN, AWS DirectConnect, or Azure ExpressRoute—to securely and reliably access the on-premises corporate data center where the PostgreSQL database resides.

Serverless job compute & Serverless SQL warehouse:
Serverless compute architecture runs entirely inside a Databricks-managed cloud account (the serverless compute plane) rather than inside the customer’s virtual network. Consequently, it does not natively support direct network-level integration (like VPNs or direct routes) to physical on-premises environments. While Databricks serverless supports Network Connectivity Configurations (NCCs) for cloud-native storage and services, it cannot naturally bridge the gap to a local, on-premises data center without complex proxy setups or data-staging workarounds.

All-purpose classic compute:
While an all-purpose classic cluster can technically establish a connection to an on-premises database via VNet injection, it is structurally designed for interactive, ad-hoc, and collaborative analysis (e.g., data scientists sharing a notebook). Utilizing an all-purpose cluster to handle an automated daily production job violates Databricks best practices, as it drastically increases costs due to idle cluster times and higher pricing tiers compared to dedicated job compute.

22. A data engineer needs to use Lakeflow Connect to incrementally ingest customer data from a supported SaaS application into Unity Catalog-governed tables. Which configuration step in Lakeflow Connect is required to achieve this ingestion?

Select a Unity Catalog connection that stores the application credentials, then configure the destination catalog and schema, and set the schedule and notifications.

Databricks Lakeflow Connect provides a fully managed, no-code/low-code experience for incrementally ingesting data from SaaS applications (like Salesforce, GitHub, ServiceNow, Confluence, etc.) directly into tables governed by Unity Catalog.

Setting up a SaaS connector follows a standard wizard-driven workflow that abstracts away manual engineering tasks:

Connection Setup: You select or create a Unity Catalog connection object that safely securely manages and stores the credentials/tokens required to authenticate with the source SaaS application’s API.
Pipeline Configuration & Target Selection: You name the ingestion pipeline and pick the target destination (the parent catalog and schema managed within Unity Catalog) where the streaming Delta tables will land.
Schedules & Notifications: You set up the orchestrating Lakeflow Job frequency (the execution schedule) to ingest new/changed records incrementally via built-in API change-tracking mechanics, and you configure email alerts for pipeline outcomes.

Manual extraction of CSV or JSON files into a volume followed by writing custom COPY INTO or Auto Loader code represents the traditional, code-heavy approach. Lakeflow Connect is explicitly designed to handle connection management, API pagination, and incremental Change Data Capture (CDC) without needing these manual interim stages or explicit file-parsing code.

Lakehouse Federation (which uses “foreign catalogs” to query external databases in real-time) is separate from Lakeflow Connect’s managed SaaS pipelines, which pull data downstream incrementally into storage rather than just federating live query access.

23. A data engineering team is managing Delta tables in Unity Catalog under the infra_catalog.ops_db schema. One of the tables, device_logs, was originally created as an external table, but the team now wants Databricks to fully manage the table lifecycle, including automatic cleanup of underlying files when the table is dropped. Which command should the team use to convert this external table into a managed table while maintaining the same table name, permissions, and history?

ALTER TABLE infra_catalog.ops_db.device_logs SET MANAGED;

This command converts an existing external Unity Catalog table into a managed table while preserving:

The table history (allowing for continuous time travel), and
The same table configurations, including table name, Unity Catalog permissions, tags, properties, and associated views. So, it avoids the manual overhead of rebuilding the table metadata.

After conversion, Databricks fully manages the table lifecycle, including automatic deletion of underlying data files when the table is dropped.