Certified Data Engineer Associate 2026

1. A data engineer has a DataFrame, salesDf, wilh a column named amount and wants to

  1. Permanently rename this column to total_amount for all future operations on salesDf
  2. In a separate query, temporarily show the column as gross_amount in the result, without changing the underlying schema

Which set of code fragments meets the requirements?

Operation 1: salesDf = salesDf.withColumnRenamed(“amount”, “total_amount”)
Operation 2: salesDf.select(col(“*”), col(“amount”).alias(“gross_amount”))

2. A data engineer builds a Lakeflow Job that ingests JSON logs from cloud storage. New files arrive at unpredictable times, and the team wants to automatically run the job as soon as new files land so the data is processed immediately. What trigger configuration should the data engineer use for this job?

File arrival trigger
In Databricks Lakeflow Jobs, a File Arrival trigger is designed to monitor a specified cloud storage path. This can be a Unity Catalog volume or an external location such as Amazon S3, Azure Storage, or Google Cloud Storage. The trigger starts a job run when new files arrive in the monitored location. This feature is useful when scheduled jobs become inefficient due to irregular data arrival patterns.

  1. Continuous trigger: Keeps a job running indefinitely by launching a new run immediately after the previous one finishes or fails. It is optimized for continuous stream processing rather than reacting dynamically to newly dropped batch files.
  2. Scheduled trigger: Runs jobs at specific, fixed intervals (e.g., every hour or daily at 12:00 AM). Because the logs arrive at completely unpredictable times, a scheduled trigger would cause latency or waste compute resources processing empty paths.
  3. Table update trigger: Listens for changes/commits made to existing Delta or Unity Catalog tables, rather than checking a raw cloud storage landing zone for new unparsed JSON log files.

3. A data engineering team builds a Lakeflow Job with an ingestion task followed by two possible execution paths: one for streaming processing and one for batch processing. They want to select the appropriate path dynamically based on a parameter processing_mode.

Use an if/else conditional task to route execution based on the processing_mode parameter.
Databricks Lakeflow Jobs explicitly provide an If/else condition task type designed to introduce boolean branching logic into your Directed Acyclic Graph (DAG) workflows.
By setting up an If/else task, you can evaluate runtime parameters (such as processing_mode) directly at the orchestration level. Based on whether the condition evaluates to true or false, Lakeflow will dynamically route the execution down the appropriate downstream path (streaming or batch), leaving the irrelevant branch untouched and saving computational resources.

4. A data engineer is trying to use Delta time travel feature to rollback a table to a previous version, but the data engineer received an error that the data files are no longer present. Which of the following commands was run on the table that caused deleting the data files?

Vacuum command
The Delta Lake vacuum command is used to clean up old data files that are no longer referenced by the Delta table. When a vacuum operation is performed, it permanently deletes the data files that are older than the retention period (default is 7 days). If a data engineer attempts to use Delta time travel to access a previous version of the table after the vacuum command has been executed, they will encounter an error indicating that the data files are no longer present, as those files have been permanently removed.

5. A data engineer is working with a PySpark DataFrame df that contains online store visits. Each row represents a customer viewing a product page, with several columns including: user_id, product_id, and timestamp. They want to ensure uniqueness at the product-view level, meaning each user should only have one record per product. Which of the following PySpark code correctly implements this logic?

1
df.dropDuplicates(["user_id", "product_id"])

The goal is to ensure uniqueness at the product-view level per user, meaning a single user should only have one recorded view for a specific product.

In PySpark, the dropDuplicates() (or its alias distinct()) method removes duplicate rows from a DataFrame. When you pass a list of column names to dropDuplicates([cols]), PySpark considers rows to be duplicates if they have identical values only in those specified columns. It then keeps the first occurrence it encounters and drops the rest.

6. “A feature that illustrates the relationship between different data assets including tables, queries, notebooks, and dashboards, enabling users to trace the origin and flow of data across the entire lakehouse platform.” Which of the following is being described in the above statement?

Unity Catalog Data Lineage

The description accurately defines the Data Lineage feature within Unity Catalog. This functionality is designed to provide transparency into data flows, allowing users to trace how datasets are created, modified, and consumed throughout the Databricks workspace. It is an essential tool for data governance, auditing, and ensuring the reliability of analytics workflows.

7. According to the Databricks Lakehouse architecture, which of the following is located in the customer’s cloud account?

Classic compute virtual machines

When the customer sets up a classic compute, the cluster virtual machines are deployed in the data plane in the customer’s cloud account.

8. A data engineer is exploring a large dataset in Databricks and executes df.summary() to better understand the data.

A DataFrame containing statistical metrics, including count, mean, standard deviation, min, max, and approximate quartile values for each column.

The df.summary() command is used for exploratory data analysis. It computes specified statistics for all numerical and string columns in a DataFrame.

When you run df.summary() without any specific arguments, it automatically calculates the following metrics for each applicable column:

  • count (number of non-null values)
  • mean
  • stddev (standard deviation)
  • min
  • 25% (approximate 1st quartile)
  • 50% (median / approximate 2nd quartile)
  • 75% (approximate 3rd quartile)
  • max

The output itself is returned as a standard Spark DataFrame, which you can then view using display(df.summary()) in a Databricks notebook.

9. A data engineering team has the following Declarative Automation Bundle (DAB) configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
variables:
 schema_name:
   default: demo_schema
 
resources:
 jobs:
   sales_job:
     name: sales_job
     tasks:
       - task_key: load_data
         notebook_task:
           notebook_path: ../src/load_data.py
           base_parameters:
             schema_name: ${var.schema_name}

A data engineer deploys the bundle using:

1
databricks bundle deploy -t dev --var="schema_name=finance_schema"

Later, the data engineer runs the job using:

1
databricks bundle run -t dev sales_job --var="schema_name=marketing_schema"

What value will be passed to the notebook parameter schema_name during the job run?

finance_schema

  1. Deployment-Time Interpretation: Bundle variables (${var.schema_name}) are deployment-time concepts in Declarative Automation Bundles (DABs). When you execute databricks bundle deploy -t dev --var="schema_name=finance_schema", the CLI completely evaluates all the custom variables and substitutes them directly into the deployed job definition configuration in the workspace.
    Therefore, the deployed job definition is explicitly written with:
    1
    2
    base_parameters:
        schema_name: finance_schema
  2. Run-Time Evaluation: When you later execute databricks bundle run, it tells Databricks to trigger the already-deployed job configuration. The --var flag is explicitly a deployment-time modifier for variable evaluation; it cannot dynamically manipulate standard bundle variable references at runtime during a bundle run.
    Because the schema_name parameter was already hardcoded as finance_schema in the workspace upon deployment, that is the value passed to the notebook.

10. A data engineering team is developing a complex ETL pipeline on Databricks. They want to ensure that their workflow configurations are version-controlled and can be deployed reliably across staging and production environments. What is the most appropriate solution to achieve this task?

Store the source code in Git folders, and deploy jobs using Databricks Asset Bundles (DAB)

Databricks Asset Bundles allow teams to define jobs and workflows declaratively in a YAML file, promoting consistency, version control, and automation. These configurations can be stored and tracked in GitHub, making deployments reproducible and reliable. Other methods rely on manual processes or ad hoc tooling and lack the benefits of structured DevOps practices.

11. A data analyst at a retail company is responsible for generating daily reports on sales performance across multiple regions and product categories. The company ingests transaction data continuously from its online stores using Lakeflow Declarative Pipelines (formerly Delta Live Tables). The analyst needs to create a relational object that can efficiently precompute business-level aggregations, such as total revenue, average order value, and units sold per category, so that downstream reporting and dashboards can access the data quickly without recalculating it every time. Which of the following objects is most suitable for this use case?

Materialized View

The most suitable object for this use case is a materialized view because it allows the data analyst to precompute and store business-level aggregations, such as total revenue, average order value, and units sold per category, so that downstream reports and dashboards can access the results quickly without recalculating them every time, unlike a temporary or standard view, which either exist only for the session or require repeated recomputation, and unlike a streaming table, which is designed for processing raw, real-time event streams rather than pre-aggregated summaries.

12. Which of the following best describes the correct format of a Databricks Asset Bundle configuration file?

A YAML file named databricks.yml that defines the bundle’s structure, including targets, resources, and configurations.

Databricks Asset Bundles use a structured YAML configuration file named databricks.yml as the primary entry point. This file defines key aspects of the bundle, such as:

  • Targets: Represent different environments (e.g., dev, staging, prod).
  • Resources: Define objects like jobs, notebooks, pipelines, and clusters.
  • Configurations: Include settings such as workspace paths, environment variables, and parameter values.

The format is designed to be declarative and human-readable, facilitating clear version control and automation within CI/CD workflows. Unlike JSON or TOML, YAML is preferred for its readability and native support for nested structures, making it ideal for complex infrastructure and deployment definitions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
bundle:
    name: my_bundle
 
targets:
    dev:
        default: true
 
    prod:
        workspace:
            host: https://<production-workspace-url>
 
resources:
    jobs:
        my-job

13. Which of the following statements best describes Auto Loader?

Auto loader monitors a source location, in which files accumulate, to identify and ingest only new arriving files with each command run. While the files that have already been ingested in previous runs are skipped.

Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage.

14. A data engineering team is ingesting 100,000 new files of small JSON files per day into a Delta table from Azure Data Lake Storage. The team requires low-latency incremental ingestion and wants to minimize storage API costs caused by frequent directory scans. Which ingestion configuration should the team prioritize?

Use Auto Loader with cloudFiles.useNotifications = true to enable event-based file detection

cloudFiles.useNotifications = true is an option in Databricks Auto Loader that switches the ingestion mechanism from directory listing to an event-driven notification system.

Instead of repeatedly scanning an ADLS folder to detect new files, Auto Loader uses Azure Event Grid notifications to learn when new files arrive.
So the flow becomes:

  • A new file lands in Azure Data Lake Storage (ADLS)
  • ADLS emits an event via Event Grid
  • Databricks Auto Loader receives the event
  • Only that file is picked up for processing

Without notifications, Auto Loader uses:

  • Directory listing mode → periodically scans folders to find new files

That works, but at very large scale (like 100,000 files/day), listing becomes:

  • slower
  • more expensive (API calls)
  • less efficient

Benefits of useNotifications = true

  • Faster ingestion latency (near real-time detection)
  • Lower storage/listing costs (fewer ADLS list operations)
  • Better scalability for very high file volumes
  • Still supports incremental processing + checkpointing
  • Works well with schema inference and evolution
  • Maintains Unity Catalog lineage when writing to governed tables

It’s ideal when:

  • You have very high file arrival rates
  • Your storage supports Event Grid integration (like ADLS Gen2)
  • You want minimal overhead and near real-time ingestion

Simple mental model

  • Without notifications → “Databricks keeps checking the folder”
  • With notifications → “Storage tells Databricks when something new arrives”

15. A data engineer uses the following SQL query:

1
GRANT MODIFY ON TABLE employees TO hr_team

Which of the following describes the ability given by the MODIFY privilege ?

It gives the ability to add, update, or delete data within the table

The MODIFY privilege grants a user the ability to add, update, or delete data from an object. It is typically assigned to users who need to make changes to existing records or insert new information into the object.