Certified Data Engineer Associate
- A data organization leader is upset about the data analysis team’s reports being different from the data engineering team’s reports. The leader believes the siloed nature of their organization’s data engineering and data analysis architectures is to blame. Which of the following describes how a data lakehouse could alleviate this issue?
Answer: Both teams would use the same source of truth of their work.
Databricks Lakehouse enables using data as the single source of truth. Duplicating data often results in data silos in organizations.
A data lakehouse is designed to unify the data engineering and data analysis architectures by integrating features of both data lakes and data warehouses. One of the key benefits of a data lakehouse is that it provides a common, centralized data repository (the “lake”) that serves as a single source of truth for data storage and analysis. This allows both data engineering and data analysis teams to work with the same consistent data sets, reducing discrepancies and ensuring that the reports generated by both teams are based on the same underlying data.
- Which of the following describes a scenario in which a data team will want to utilize cluster pools?
Answer: An automated report needs to be refreshed as quickly as possible.
Using cluster pools reduces the cluster startup time. So in this case, the reports can be refreshed quickly and not having to wait long for the cluste to start.
- Which of the following is hosted completely in the control plane of the classic Databricks architecture?
Options:
- Worker node
- JDBC data source
- Databricks web application
- Databricks Filesystem
- Driver node
Answer: Databricks web application
In the classic Databricks architecture, the control plane includes components like the Databricks web application, the Databricks REST API, and the Databricks Workspace. These components are responsible for managing and controlling the Databricks environment, including cluster provisioning, notebook management, access control, and job scheduling.
The other options, such as worker nodes, JDBC data sources, Databricks Filesystem (DBFS), and driver nodes, are typically part of the data plane or the execution environment, which is separate from the control plane. Worker nodes are responsible for executing tasks and computations, JDBC data sources are used to connect to external databases, DBFS is a distributed file system for data storage, and driver nodes are responsible for coordinating the execution of Spark jobs.
- Which of the following benets of using the Databricks Lakehouse Platform is provided by Delta Lake?
Answer: The ability to support batch and streaming workloads.
Delta Lake is a key component of the Databricks Lakehouse Platform that provides several benefits, and one of the most significant benefits is its ability to support both batch and streaming workloads seamlessly. Delta Lake allows you to process and analyze data in real-time (streaming) as well as in batch, making it a versatile choice for various data processing needs.
- Which of the following describes the storage organization of a Delta table?
Answer: Delta tables are stored in a collection of files that contain data, history, metadata, and other attributes.
Delta tables are organized as a collection of files that include not only the data itself but also the history of changes, metadata about the table, and other attributes. This organization allows Delta Lake to provide features such as ACID transactions, time travel, and schema enforcement, making it easier to manage and query data efficiently.
- Which of the following code blocks will remove the rows where the value in column age is greater than 25 from the existing Delta table
my_table and save the updated table?
Answer: DELETE FROM my_table WHERE age > 25;
- A data engineer has realized that they made a mistake when making a daily update to a table. They need to use Delta time travel to restore the table to a version that is 3 days old. However, when the data engineer attempts to time travel to the older version, they are unable to restore the data because the data files have been deleted. Which of the following explains why the data files are no longer present?
Answer: The VACUUM command was run on the table
There is no DELETE HISTORY command in Databricks
VACCUMcommand can remove history and we can also specify the retention period with VACCUM Command. Default Retention period is 7 days. To allow changing the default retention period you can run the following command:ALTER TABLE your_table SET TBLPROPERTIES ('delta.retentionDurationCheck.enabled' = 'true');The
RESTOREcommand:DESCRIBE HISTORY my_table;can be used to view the history of a Delta table and identify the version to which you want to restore. However, if the data files associated with that version have been deleted (e.g., through a VACUUM command), you will not be able to restore the table to that version, as the necessary data files are no longer available.RESTORE TABLE my_table TO VERSION AS OF 10;can be used to restore the table to a specific version, but it will fail if the data files for that version have been deleted.RESTORE TABLE my_table TO TIMESTAMP AS OF '2026-06-08 10:00:00';can be used to restore the table to a specific version, but it will fail if the data files for that version have been deleted.
Check the old version of the table: SELECT * FROM my_table VERSION AS OF 10; or SELECT * FROM my_table TIMESTAMP AS OF '2026-06-08 10:00:00';
The
VACCUMNcommand:VACUUM my_table;can be used to remove old data files and history from a Delta table. If theVACUUMcommand was run on the table, it may have removed the data files associated with older versions of the table, making it impossible to restore to those versions using time travel. Default Retention period is 7 days.
VACUUM my_table RETAIN 168 HOURS; can be used to retain the data files associated with older versions of the table for a specified period of time.VACUUM my_table RETAIN 720 HOURS; can be used to retain the data files associated with older versions of the table for a specified period of time.
Dry run: VACUUM my_table RETAIN 168 HOURS DRY RUN; can be used to see which files would be removed without actually deleting them.
Common use:
1 | -- Step 1: check the history of the table to identify the version to restore |
Clean up:
1 | -- Step 1: confirm the files to be removed before actually deleting them |
- Which of the following Git operations must be performed outside of Databricks Repos?
Answer: git clone
- Which of the following data lakehouse features results in improved data quality over a traditional data lake?
Answer: A data lakehouse supports ACID-compliant transactions.
ACID (Atomicity, Consistency, Isolation, Durability) compliance ensures that transactions are processed reliably and consistently, which is crucial for maintaining data integrity and quality. It helps in preventing data inconsistencies or errors that might occur during concurrent transactions or data operations. By supporting ACID-compliant transactions, a data lakehouse can enhance data quality by providing mechanisms to ensure the reliability and consistency of data operations, which is a notable improvement over traditional data lakes that might lack such transactional capabilities.
- A data engineer needs to determine whether to use the built-in Databricks Notebooks versioning or version their project using Databricks Repos. Which of the following is an advantage of using Databricks Repos over the Databricks Notebooks versioning?
Answer: Databricks Repos supports the use of multiple branches
- A data engineer has left the organization. The data team needs to transfer ownership of the data engineer’s Delta tables to a new data engineer. The new data engineer is the lead engineer on the data team. Assuming the original data engineer no longer has access, which of the following individuals must be the one to transfer ownership of the Delta tables in Data Explorer?
Answer: Workspace administrator.
- A data analyst has created a Delta table sales that is used by the entire data analysis team. They want help from the data engineering team to implement a series of tests to ensure the data is clean. However, the data engineering team uses Python for its tests rather than SQL. Which of the following commands could the data engineering team use to access sales in PySpark?
Answer: spark.table(“sales”)
- Which of the following commands will return the location of database customer360?
Answer: DESCRIBE DATABASE customer360;
- A data engineer wants to create a new table containing the names of customers that live in France. They have written the following command:
`` A senior data engineer mentions that it is organization policy to include a table property indicating that the new table includes personally identiable information (PII). Which of the following lines of code fills in the above blank to successfully complete the task?
Answer: COMMENT “Contains PII”
CREATE TABLE my_table (id INT COMMENT 'Unique Identification Number', name STRING COMMENT 'PII', age INT COMMENT 'PII') TBLPROPERTIES ('contains_pii'=True)
- Which of the following benets is provided by the array functions from Spark SQL?
Answer: An ability to work with complex, nested data ingested from JSON les
- Which of the following commands can be used to write data into a Delta table while avoiding the writing of duplicate records?
Answer: MERGE INTO my_table USING delta_table ON my_table.id = delta_table.id WHEN MATCHED THEN UPDATE SET my_table.name = delta_table.name.
- A data engineer needs to apply custom logic to string column city in table stores for a specic use case. In order to apply this custom logic at scale, the data engineer wants to create a SQL user-dened function (UDF). Which of the following code blocks creates this SQL UDF?
Answer:
1 | Create function combine_nyc(city STRING) RETURNS STRING |
- A data analyst has a series of queries in a SQL program. The data analyst wants this program to run every day. They only want the final query in the program to run on Sundays. They ask for help from the data engineering team to complete this task. Which of the following approaches could be used by the data engineering team to complete this task?
Answer: They could wrap the queries using PySpark and use Python’s control ow system to determine when to run the nal query.
- A data engineer runs a statement every day to copy the previous day’s sales into the table transactions. Each day’s sales are in their own file in the location “/transactions/raw”. Today, the data engineer runs the following command to complete this task:
1 | COPY INTO transactions FROM "/transations/raw" FILEFORMAT = PARQUET; |
After running the command today, the data engineer notices that the number of records in table transactions has not changed. Which of the following describes why the statement might not have copied any new records into the table?
Answer: The previous day’s file has already been copied into the table.
In Databricks, COPY INTO is idempotent. It keeps track of which files have already been loaded into the target Delta table. So if the data engineer runs COPY INTO transactions FROM “/transactions/raw” again, Databricks will skip files that were already copied.
- A data engineer needs to create a table in Databricks using data from their organization’s existing SQLite database. They run the following command:
1 | CREATE TABLE jdbc_customer360 USING __ OPTIONS (url="jdbc:sqlite:/customer360.db", dbtable="customer360"); |
Which of the following lines of code lls in the above blank to successfully complete the task?
Answer: org.apache.spark.sql.jdbc
- A data engineering team has two tables. The rst table march_transactions is a collection of all retail transactions in the month of March. The second table april_transactions is a collection of all retail transactions in the month of April. There are no duplicate records between the tables. Which of the following commands should be run to create a new table all_transactions that contains all records from march_transactions and april_transactions without duplicate records?
Answer:
1 | CREATE TABLE all_transactions AS |
The
UNIONcommand can be used to combine the results of two or more SELECT statements into a single result set.
TheUNION ALLcommand can be used to combine the results of two or more SELECT statements into a single result set, even if there are duplicate records in the result sets.
- A data engineer only wants to execute the nal block of a Python program if the Python variable day_of_week is equal to 1 and the Python variable review_period is True. Which of the following control flow statements should the data engineer use to begin this conditionally executed code block?
Answer: if day_of_week == 1 and review_period:
- A data engineer is attempting to drop a Spark SQL table my_table. The data engineer wants to delete all table metadata and data. They run the following command:
1 | DROP TABLE IF EXISTS my_table; |
While the object no longer appears when they run SHOW TABLES, the data files still exist. Which of the following describes why the data files still exist and the metadata files were deleted?
Answer: The table was external.
The
DROP TABLEcommand in Spark SQL behaves differently based on whether the table is internal or external. For an internal table, both the metadata and the data files are deleted when the table is dropped. However, for an external table, only the metadata is deleted, and the data files remain intact. This allows for the possibility of re-creating the external table later without losing the underlying data. In this case, since the data files still exist after dropping the table, it indicates that the table was defined as an external table.
- A data engineer wants to create a data entity from a couple of tables. The data entity must be used by other data engineers in other sessions. It also must be saved to a physical location. Which of the following data entities should the data engineer create?
Answer: Table.
To create a data entity that can be used by other data engineers in other sessions and must be saved to a physical location, you should create a table. Tables in a database are physical storage structures that hold data, and they can be accessed and shared by multiple users and sessions. By creating a table, you provide a permanent and structured storage location for the data entity that can be used across different sessions and by other users as needed.
Options like databases (A) can be used to organize tables, views (C) can provide virtual representations of data, and temporary views (D) are temporary in nature and don’t save data to a physical location. Functions (B) are typically used for processing data or performing calculations, not for storing data.
- A data engineer is maintaining a data pipeline. Upon data ingestion, the data engineer notices that the source data is starting to have a lower level of quality. The data engineer would like to automate the process of monitoring the quality level. Which of the following tools can the data engineer use to solve this problem?
Answer: Delta Live Tables.
Delta Live Tables is a tool provided by Databricks that can help data engineers automate the monitoring of data quality. It is designed for managing data pipelines, monitoring data quality, and automating workflows. With Delta Live Tables, you can set up data quality checks and alerts to detect issues and anomalies in your data as it is ingested and processed in real-time. It provides a way to ensure that the data quality meets your desired standards and can trigger actions or notifications when issues are detected.
- A Delta Live Table pipeline includes two datasets dened using STREAMING LIVE TABLE. Three datasets are dened against Delta Lake table sources using LIVE TABLE.
The table is congured to run in Production mode using the Continuous Pipeline Mode. Assuming previously unprocessed data exists and all denitions are valid, what is the expected outcome after clicking Start to update the pipeline?
Answer: All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will be deployed for the update and terminated when the pipeline is stopped.
Continuous Pipeline Mode in Production mode implies that the pipeline continuously processes incoming data updates at set intervals, ensuring the datasets are kept up-to-date as new data arrives.
Since the pipeline is set to Continuous Pipeline Mode, it will keep running and updating the datasets until it is manually shut down. The compute resources are allocated dynamically to process and update the datasets as needed, and they will be terminated when the pipeline is stopped or shut down. This mode allows for real-time or near-real-time updates to the datasets from the streaming/live tables, ensuring that the data remains current and reflects the changes occurring in the data sources.
- In order for Structured Streaming to reliably track the exact progress of the processing so that it can handle any kind of failure by restarting and/or reprocessing, which of the following two approaches is used by Spark to record the offset range of the data being processed in each trigger?
Answer: Checkpointing and Write-ahead Logs
The engine uses checkpointing and write-ahead logs to record the offset range of the data being processed in each trigger. The streaming sinks are designed to be idempotent for handling reprocessing. Together, using replayable sources and idempotent sinks, Structured Streaming can ensure end-to-end exactly-once semantics under any failure.
- Which of the following describes the relationship between Gold tables and Silver tables?
Answer: Gold tables are more likely to contain aggregations than Silver tables.
In some data processing pipelines, especially those following a typical “Bronze-Silver-Gold” data lakehouse architecture, Silver tables are often considered a more refined version of the raw or Bronze data. Silver tables may include data cleansing, schema enforcement, and some initial transformations. Gold tables, on the other hand, typically represent a stage where data is further enriched, aggregated, and processed to provide valuable insights for analytical purposes. This could indeed involve more aggregations compared to Silver tables.
- Which of the following describes the relationship between Bronze tables and raw data?
Answer: Bronze tables contain raw data with a schema applied.
Bronze tables are basically raw ingested data, often with schema borrowed from the original data source or table.
- Which of the following tools is used by Auto Loader process data incrementally?
Answer: Spark Structured Streaming.
The Auto Loader process in Databricks is typically used in conjunction with Spark Structured Streaming to process data incrementally. Spark Structured Streaming is a real-time data processing framework that allows you to process data streams incrementally as new data arrives. The Auto Loader is a feature in Databricks that works with Structured Streaming to automatically detect and process new data files as they are added to a specified data source location. It allows for incremental data processing without the need for manual intervention.
- A data engineer has congured a Structured Streaming job to read from a table, manipulate the data, and then perform a streaming write into a new table. The cade block used by the data engineer is below:
1 | ( |
If the data engineer only wants the query to execute a micro-batch to process data every 5 seconds, which of the following lines of code
should the data engineer use to ll in the blank?
Answer: trigger(processingTime=”5 seconds”)
- A dataset has been dened using Delta Live Tables and includes an expectations clause:
CONSTRAINT valid_timestamp EXPECT (timestamp > '2020-01-01') ON VIOLATION DROP ROWWhat is the expected behavior when a batch of data containing data that violates these constraints is processed?
Answer: Records that violate the expectation are dropped from the target dataset and recorded as invalid in the event log.
The defined expectation specifies that if the timestamp is not greater than ‘2020-01-01’, the row will be considered in violation of the constraint. The ON VIOLATION DROP ROW clause states that rows that violate the constraint will be dropped from the target dataset. Additionally, the expectation clause will log these violations in the event log, indicating which records did not meet the specified constraint criteria.
This behavior ensures that the rows failing the defined constraint are not included in the target dataset and are logged as invalid in the event log for reference or further investigation, maintaining data integrity within the dataset based on the specified constraints.
- Which of the following describes when to use the CREATE STREAMING LIVE TABLE (formerly CREATE INCREMENTAL LIVE TABLE) syntax over the CREATE LIVE TABLE syntax when creating Delta Live Tables (DLT) tables using SQL?
Answer: CREATE STREAMING LIVE TABLE should be used when data needs to be processed incrementally.
The CREATE STREAMING LIVE TABLE syntax is used when you want to create Delta Live Tables (DLT) tables that are designed for processing data incrementally. This is typically used when your data pipeline involves streaming or incremental data updates, and you want the table to stay up to date as new data arrives. It allows you to define tables that can handle data changes incrementally without the need for full table refreshes.
So, option B correctly describes when to use CREATE STREAMING LIVE TABLE over CREATE LIVE TABLE in the context of Delta Live Tables.
- A data engineer is designing a data pipeline. The source system generates files in a shared directory that is also used by other processes. As a result, the files should be kept as is and will accumulate in the directory. The data engineer needs to identify which files are new since the previous run in the pipeline, and set up the pipeline to only ingest those new files with each run. Which of the following tools can the data engineer use to solve this problem?
Answer: Auto Loader
Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage without any additional setup.
- Which of the following Structured Streaming queries is performing a hop from a Silver table to a Gold table?
1 | (spark.table("sales") |
- A data engineer has three tables in a Delta Live Tables (DLT) pipeline. They have congured the pipeline to drop invalid records at each table. They notice that some data is being dropped due to quality concerns at some point in the DLT pipeline. They would like to determine at which table in their pipeline the data is being dropped. Which of the following approaches can the data engineer take to identify the table that is dropping the records?
Answer: They can navigate to the DLT pipeline page, click on each table, and view the data quality statistics.
To identify the table in a Delta Live Tables (DLT) pipeline where data is being dropped due to quality concerns, the data engineer can navigate to the DLT pipeline page, click on each table in the pipeline, and view the data quality statistics. These statistics often include information about records dropped, violations of expectations, and other data quality metrics. By examining the data quality statistics for each table in the pipeline, the data engineer can determine at which table the data is being dropped.
- A data engineer has a single-task Job that runs each morning before they begin working. After identifying an upstream data issue, they need to set up another task to run a new notebook prior to the original task. Which of the following approaches can the data engineer use to set up the new task?
Answer: They can create a new task in the existing Job and then add it as a dependency of the original task.
It seems there is some confusion on what dependency means in this case. Option B is correct because adding the new task as a dependency of the original task means that the new task will run BEFORE the original task, which is the goal defined in the question.
- An engineering manager wants to monitor the performance of a recent project using a Databricks SQL query. For the rst week following the project’s release, the manager wants the query results to be updated every minute. However, the manager is concerned that the compute resources used for the query will be left running and cost the organization a lot of money beyond the rst week of the project’s release. Which of the following approaches can the engineering team use to ensure the query does not cost the organization any money beyond the rst week of the project’s release?
Answer: They can set the query’s refresh schedule to end on a certain date in the query scheduler.
- A data analysis team has noticed that their Databricks SQL queries are running too slowly when connected to their always-on SQL endpoint. They claim that this issue is present when many members of the team are running small queries simultaneously. They ask the data engineering team for help. The data engineering team notices that each of the team’s queries uses the same SQL endpoint. Which of the following approaches can the data engineering team use to improve the latency of the team’s queries?
Answer: They can increase the maximum bound of the SQL endpoint’s scaling range.
- A data engineer wants to schedule their Databricks SQL dashboard to refresh once per day, but they only want the associated SQL endpoint to be running when it is necessary.
Which of the following approaches can the data engineer use to minimize the total running time of the SQL endpoint used in the refresh schedule of their dashboard?
Answer: They can turn on the Auto Stop feature for the SQL endpoint.
The data engineer can use the Auto Stop feature to minimize the total running time of the SQL endpoint used in the refresh schedule of their dashboard. The Auto Stop feature allows the SQL endpoint to automatically shut down when there are no active connections, which will minimize the total running time of the SQL endpoint. By scheduling the dashboard to refresh once per day, the SQL endpoint will only be running for a short period of time each day, which will minimize the total running time and reduce costs.
- A data engineer has been using a Databricks SQL dashboard to monitor the cleanliness of the input data to an ELT job. The ELT job has its Databricks SQL query that returns the number of input records containing unexpected NULL values. The data engineer wants their entire team to be notied via a messaging webhook whenever this value reaches 100.
Which of the following approaches can the data engineer use to notify their entire team via a messaging webhook whenever the number of NULL values reaches 100?
Answer: They can set up an Alert with a new webhook alert destination.
- A single Job runs two notebooks as two separate tasks. A data engineer has noticed that one of the notebooks is running slowly in the Job’s current run. The data engineer asks a tech lead for help in identifying why this might be the case. Which of the following approaches can the tech lead use to identify why the notebook is running slowly as part of the Job?
Answer: They can navigate to the Runs tab in the Jobs UI and click on the active run to review the processing notebook.
The tech lead can navigate to the Runs tab in the Jobs UI and click on the active run to review the processing notebook (Option C). This will allow them to inspect the details of the job run, including the duration of each task, which can help identify potential performance issues. There could be several reasons why a notebook is running slowly as part of a job. For instance, there might be a delay when the job cluster has to be spun up, or the table gets delta cached in memory and copies of files will be stored on local node’s storage. Even certain operations like pandas UDFs can be slow.
Please note that the exact process may vary depending on the specific configurations and permissions set up in your workspace. It’s always a good idea to consult with your organization’s IT or data governance team to ensure the correct procedures are followed.
- A data engineer has a Job with multiple tasks that runs nightly. Each of the tasks runs slowly because the clusters take a long time to start. Which of the following actions can the data engineer perform to improve the start up time for the clusters used for the Job?
Answer: They can use clusters that are from a cluster pool.
Cluster pools are a way to pre-provision clusters that are ready to use. This can reduce the start up time for clusters, as they do not have to be created from scratch.
All-purpose clusters are not pre-provisioned, so they will take longer to start up. Jobs clusters are a type of cluster pool, but they are not the best option for this use case. Jobs clusters are designed for long-running jobs, and they can be more expensive than other types of cluster pools. Single-node clusters are the smallest type of cluster, and they will start up the fastest. However, they may not be powerful enough to run the Job’s tasks.
Autoscaling clusters can scale up or down based on demand. This can help to improve the start up time for clusters, as they will only be created when they are needed. However, autoscaling clusters can also be more expensive than other types of cluster pool
- A new data engineering team team. has been assigned to an ELT project. The new data engineering team will need full privileges on the database customers to fully manage the project. Which of the following commands can be used to grant full permissions on the database to the new data engineering team?
Answer: GRANT ALL PRIVILEGES ON DATABASE customers TO team;
- A new data engineering team has been assigned to work on a project. The team will need access to database customers in order to see what tables already exist. The team has its own group team. Which of the following commands can be used to grant the necessary permission on the entire database to the new team?
Answer: GRANT USAGE ON DATABASE customers TO team;
- A data engineer is running code in a Databricks Repo that is cloned from a central Git repository. A colleague of the data engineer informs them that changes have been made and synced to the central Git repository. The data engineer now needs to sync their Databricks Repo to get the changes from the central Git repository. Which of the following Git operations does the data engineer need to run to accomplish this task?
Answer: Pull
- Which of the following is a benet of the Databricks Lakehouse Platform embracing open source technologies?
Answer: Avoiding vendor lock-in.
E is correct as open-source is opposite of proprietary technology, so not being a proprietary means it is free of vendor lock in, if that makes sense.
- A data engineer needs to use a Delta table as part of a data pipeline, but they do not know if they have the appropriate permissions. In which of the following locations can the data engineer review their permissions on the table?
Answer: Data Explorer
- Which of the following describes a scenario in which a data engineer will want to use a single-node cluster?
Answer: When they are working interactively with a small amount of data
- A data engineer has been given a new record of data:Which of the following SQL commands can be used to append the new record to an existing Delta table my_table?
1
2
3id STRING = 'a1'
rank INTEGER = 6
rating FLOAT = 9.4
Answer: INSERT INTO my_table VALUES (‘a1’, 6, 9.4)

