Use Databricks Certified Data Engineer Professional Dumps to Obtain Databricks Certification

Category:

Comments:

0 Comments

Post Date:

October 18, 2023

If you're serious about obtaining the Databricks Certification certification, practicing with Databricks Databricks Certified Data Engineer Professional dumps questions is a must. By doing so, you'll become familiar with the exam format, identify knowledge gaps, develop time management skills, boost your confidence, and increase your chances of success. So, start practicing Databricks Certified Data Engineer Professional exam dumps today and give yourself the best chance of passing the Databricks Certified Data Engineer Professional exam! Test free Databricks Certified Data Engineer Professional Exam Databricks Certified Data Engineer Professional exam dumps below.

Page 1 of 11

1. The data architect has mandated that all tables in the Lakehouse should be configured as external Delta Lake tables.

Which approach will ensure that this requirement is met?

Whenever a database is being created, make sure that the LOCATION keyword is used

When configuring an external data warehouse for all table storage, leverage Databricks for all EL

Whenever a table is being created, make sure that the LOCATION keyword is used.

When tables are created, make sure that the EXTERNAL keyword is used in the CREATE TABLE statement.

When the workspace is being configured, make sure that external cloud object storage has been mounted.

2. A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure.

The silver_device_recordings table will be used downstream to power several production monitoring dashboards and a production model. At present, 45 of the 100 fields are being used in at least one of these applications.

The data engineer is trying to determine the best approach for dealing with schema declaration given the highly-nested structure of the data and the numerous fields.

Which of the following accurately presents information about Delta Lake and Databricks that may impact their decision-making process?

The Tungsten encoding used by Databricks is optimized for storing string data; newly-added native support for querying JSON strings means that string types are always most efficient.

Because Delta Lake uses Parquet for data storage, data types can be easily evolved by just modifying file footer information in place.

Human labor in writing code is the largest cost associated with data engineering workloads; as such, automating table declaration logic should be a priority in all migration workloads.

Because Databricks will infer schema using types that allow all observed data to be processed, setting types manually provides greater assurance of data quality enforcement.

Schema inference and evolution on Databricks ensure that inferred types will always accurately match the data types used by downstream systems.

3. 1.You were asked to setup a new all-purpose cluster, but the cluster is unable to start which of the following steps do you need to take to identify the root cause of the issue and the reason why the cluster was unable to start?

Check the cluster driver logs

Check the cluster event logs (Correct)

Workspace logs

Storage account

Data plane

4. Which statement regarding stream-static joins and static Delta tables is correct?

Each microbatch of a stream-static join will use the most recent version of the static Delta table as of each microbatch.

Each microbatch of a stream-static join will use the most recent version of the static Delta table as of the job's initialization.

The checkpoint directory will be used to track state information for the unique keys present in the join.

Stream-static joins cannot use static Delta tables because of consistency issues.

The checkpoint directory will be used to track updates to the static Delta table.

5. Which of the following describes a benefit of a data lakehouse that is unavailable in a traditional data

warehouse?

A data lakehouse couples storage and compute for complete control

A data lakehouse provides a relational system of data management

A data lakehouse utilizes proprietary storage formats for data

A data lakehouse enables both batch and streaming analytics

A data lakehouse captures snapshots of data for version control purposes

6. You noticed that colleague is manually copying the notebook with _bkp to store the previous versions, which of the following feature would you recommend instead.

Databricks notebooks support change tracking and versioning

Databricks notebooks should be copied to a local machine and setup source control locally to version the notebooks

Databricks notebooks can be exported into dbc archive files and stored in data lake

Databricks notebook can be exported as HTML and imported at a later time

7. What is the purpose of a silver layer in Multi hop architecture?

Replaces a traditional data lake

Efficient storage and querying of full and unprocessed history of data

A schema is enforced, with data quality checks.

Refined views with aggregated data

Optimized query performance for business-critical data

8. Which of the following functions can be used to convert JSON string to Struct data type?

TO_STRUCT (json value)

FROM_JSON (json value)

FROM_JSON (json value, schema of json)

CONVERT (json value, schema of json)

CAST (json value as STRUCT)

9. Which of the following developer operations in the CI/CD can only be implemented through a GIT provider when using Databricks Repos.

Trigger Databricks Repos pull API to update the latest version

Commit and push code

Create and edit code

Create a new branch

Pull request and review process

10. The data engineering team maintains the following code:

Assuming that this code produces logically correct results and the data in the source tables has been de-duplicated and validated, which statement describes what will occur when this code is executed?

A batch job will update the enriched_itemized_orders_by_account table, replacing only those rows that have different values than the current version of the table, using accountID as the primary key.

The enriched_itemized_orders_by_account table will be overwritten using the current valid version of data in each of the three tables referenced in the join logic.

An incremental job will leverage information in the state store to identify unjoined rows in the source tables and write these rows to the enriched_iteinized_orders_by_account table.

An incremental job will detect if new rows have been written to any of the source tables; if new rows are detected, all results will be recalculated and used to overwrite the enriched_itemized_orders_by_account table.

No computation will occur until enriched_itemized_orders_by_account is queried; upon query materialization, results will be calculated using the current valid version of data in each of the three tables referenced in the join logic.

Page 2 of 11

11. You were asked to identify number of times a temperature sensor exceed threshold temperature (100.00) by each device, each row contains 5 readings collected every 5 minutes, fill in the blank with the appropriate functions.

Schema: deviceId INT, deviceTemp ARRAY, dateTimeCollected TIMESTAMP

SELECT deviceId, __ (__ (__(deviceTemp], i -> i > 100.00)))

FROM devices

GROUP BY deviceId

SUM, COUNT, SIZE

SUM, SIZE, SLICE

SUM, SIZE, ARRAY_CONTAINS

SUM, SIZE, ARRAY_FILTER

SUM, SIZE, FILTER

12. Which of the following SQL commands are used to append rows to an existing delta table?

APPEND INTO DELTA table_name

APPEND INTO table_name

COPY DELTA INTO table_name

INSERT INTO table_name

UPDATE table_name

13. A new user who currently does not have access to the catalog or schema is requesting access to the customer table in sales schema, but the customer table contains sensitive information, so you have decided to create view on the table excluding columns that are sensitive and granted access to the view GRANT SELECT ON view_name to [email protected] but when the user tries to query the view, gets the error view does not exist.

What is the issue preventing user to access the view and how to fix it?

User requires SELECT on the underlying table

User requires to be put in a special group that has access to PII data

User has to be the owner of the view

User requires USAGE privilege on Sales schema

User needs ADMIN privilege on the view

14. The team has decided to take advantage of table properties to identify a business owner for each table, which of the following table DDL syntax allows you to populate a table property identifying the business owner of a table

CREATE TABLE inventory (id INT, units FLOAT)

SET TBLPROPERTIES business_owner = 'supply chain' CREATE TABLE inventory (id INT, units FLOAT)

TBLPROPERTIES (business_owner = 'supply chain')

CREATE TABLE inventory (id INT, units FLOAT) SET (business_owner = ‘supply chain’)

CREATE TABLE inventory (id INT, units FLOAT) SET PROPERTY (business_owner = ‘supply chain’)

CREATE TABLE inventory (id INT, units FLOAT) SET TAG (business_owner = ‘supply chain’)

15. An upstream source writes Parquet data as hourly batches to directories named with the current date.

A nightly batch job runs the following code to ingest all data from the previous day as indicated by the date variable:

Assume that the fields customer_id and order_id serve as a composite key to uniquely identify each order.

If the upstream system is known to occasionally produce duplicate entries for a single order hours apart, which statement is correct?

Each write to the orders table will only contain unique records, and only those records without duplicates in the target table will be written.

Each write to the orders table will only contain unique records, but newly written records may have duplicates already present in the target table.

Each write to the orders table will only contain unique records; if existing records with the same key are present in the target table, these records will be overwritten.

Each write to the orders table will only contain unique records; if existing records with the same key are present in the target table, the operation will fail.

Each write to the orders table will run deduplication over the union of new and existing records, ensuring no duplicate records are present.

16. A data engineer is using a Databricks SQL query to monitor the performance of an ELT job. The ELT job is triggered by a specific number of input records being ready to process. The Databricks SQL query returns the number of minutes since the job’s most recent runtime.

Which of the following approaches can enable the data engineering team to be notified if the ELT job has not been run in an hour?

They can set up an Alert for the accompanying dashboard to notify them if the returned value is greater than 60.

They can set up an Alert for the query to notify when the ELT job fails.

They can set up an Alert for the accompanying dashboard to notify when it has not refreshed in 60 minutes.

They can set up an Alert for the query to notify them if the returned value is greater than 60.

This type of alert is not possible in Databricks

17. return x

18. You are tasked to set up a set notebook as a job for six departments and each department can run the task parallelly, the notebook takes an input parameter dept number to process the data by department, how do you go about to setup this up in job?

Use a single notebook as task in the job and use dbutils.notebook.run to run each note-book with parameter in a different cell

A task in the job cannot take an input parameter, create six notebooks with hardcoded dept number and setup six tasks with linear dependency in the job

A task accepts key-value pair parameters, creates six tasks pass department number as parameter foreach task with no dependency in the job as they can all run in parallel. (Correct)

A parameter can only be passed at the job level, create six jobs pass department number to each job with linear job dependency

A parameter can only be passed at the job level, create six jobs pass department number to each job with no job dependency

19. How do you check the location of an existing schema in Delta Lake?

Run SQL command SHOW LOCATION schema_name

Check unity catalog UI

Use Data explorer

Run SQL command DESCRIBE SCHEMA EXTENDED schema_name E Schemas are internally in-store external hive meta stores like MySQL or SQL Server

20. Why does AUTO LOADER require schema location?

Schema location is used to store user provided schema

Schema location is used to identify the schema of target table

AUTO LOADER does not require schema location, because its supports Schema evolution

Schema location is used to store schema inferred by AUTO LOADER

Schema location is used to identify the schema of target table and source table

Page 3 of 11

21. table(table_name))

format, checkpointlocation, schemalocation, overwrite

cloudfiles.format, checkpointlocation, cloudfiles.schemalocation, overwrite

cloudfiles.format, cloudfiles.schemalocation, checkpointlocation, mergeSchema

cloudfiles.format, cloudfiles.schemalocation, checkpointlocation, append

cloudfiles.format, cloudfiles.schemalocation, checkpointlocation, overwrite

22. A team member is leaving the team and he/she is currently the owner of the few tables, instead of transferring the ownership to a user you have decided to transfer the ownership to a group so in the future anyone in the group can manage the permissions rather than a single individual, which of the following commands help you accomplish this?

ALTER TABLE table_name OWNER to 'group'

TRANSFER OWNER table_name to 'group'

GRANT OWNER table_name to 'group'•

ALTER OWNER ON table_name to 'group'

GRANT OWNER On table_name to 'group'

23. A data engineer is designing a data pipeline. The source system generates files in a shared directory that is also used by other processes. As a result, the files should be kept as is and will accumulate in the directory. The data engineer needs to identify which files are new since the previous run in the pipeline, and set up the pipeline to only ingest those new files with each run.

Which of the following tools can the data engineer use to solve this problem?

Unity Catalog

Auto Loader

Data Explorer

Delta Lake

Databricks SQL

24. Which of the following benefits does Delta Live Tables provide for ELT pipelines over standard data pipelines that utilize Spark and Delta Lake on Databricks?

The ability to write pipelines in Python and/or SQL

The ability to declare and maintain data table dependencies

The ability to automatically scale compute resources

The ability to access previous versions of data tables

The ability to perform batch and streaming queries

25. What type of table is created when you create delta table with below command?

CREATE TABLE transactions USING DELTA LOCATION "DBFS:/mnt/bronze/transactions"

Managed delta table

External table

Managed table

Temp table

Delta Lake table

26. Which of the following technique can be used to implement fine-grained access control to rows and columns of the Delta table based on the user's access?

Use Unity catalog to grant access to rows and columns

Row and column access control lists

Use dynamic view functions

Data access control lists

Dynamic Access control lists with Unity Catalog

27. Which configuration parameter directly affects the size of a spark-partition upon ingestion of data into Spark?

spark.sql.files.maxPartitionBytes

spark.sql.autoBroadcastJoinThreshold

spark.sql.files.openCostInBytes

spark.sql.adaptive.coalescePartitions.minPartitionNum

spark.sql.adaptive.advisoryPartitionSizeInBytes

28. A user new to Databricks is trying to troubleshoot long execution times for some pipeline logic they are working on. Presently, the user is executing code cell-by-cell, using display() calls to confirm code is producing the logically correct results as new transformations are added to an operation. To get a measure of average time to execute, the user is running each cell multiple times interactively.

Which of the following adjustments will get a more accurate measure of how code is likely to perform in production?

Scala is the only language that can be accurately tested using interactive notebooks; because the best performance is achieved by using Scala code compiled to JARs, all PySpark and Spark SQL logic should be refactored.

The only way to meaningfully troubleshoot code execution times in development notebooks Is to use production-sized data and production-sized clusters with Run All execution.

Production code development should only be done using an IDE; executing code against a local build of open source Spark and Delta Lake will provide the most accurate benchmarks for how code will perform in production.

Calling display() forces a job to trigger, while many transformations will only add to the logical query plan; because of caching, repeated execution of the same logic does not provide meaningful results.

The Jobs UI should be leveraged to occasionally run the notebook as a job and track execution time during incremental code development because Photon can only be enabled on clusters launched for scheduled jobs.

29. A junior data engineer seeks to leverage Delta Lake's Change Data Feed functionality to create a Type 1 table representing all of the values that have ever been valid for all rows in a bronze table created with the property delta.enableChangeDataFeed = true.

They plan to execute the following code as a daily job:

Which statement describes the execution and results of running the above query multiple times?

Each time the job is executed, newly updated records will be merged into the target table, overwriting previous values with the same primary keys.

Each time the job is executed, the entire available history of inserted or updated records will be appended to the target table, resulting in many duplicate entries.

Each time the job is executed, the target table will be overwritten using the entire history of inserted or updated records, giving the desired result.

Each time the job is executed, the differences between the original and current versions are calculated; this may result in duplicate entries for some records.

Each time the job is executed, only those records that have been inserted or updated since the last execution will be appended to the target table, giving the desired result.

30. Which of the following operations are not supported on a streaming dataset view?

spark.readStream.format("delta").table("sales").createOrReplaceTempView("streaming_vie w")

SELECT sum(unitssold) FROM streaming_view

SELECT max(unitssold) FROM streaming_view

SELECT id, sum(unitssold) FROM streaming_view GROUP BY id ORDER BY id

SELECT id, count(*) FROM streaming_view GROUP BY id

SELECT * FROM streadming_view ORDER BY id

Page 4 of 11

31. spark.sql("SELECT * FROM sales WHERE orderDate = order_date")

32. Newly joined data analyst requested read-only access to tables, assuming you are owner/admin which section of Databricks platform is going to facilitate granting select access to the user

Admin console

User settings

Data explorer

Azure Databricks control pane IAM

Azure RBAC

33. Which of the following SQL command can be used to insert or update or delete rows based on a condition to check if a row(s) exists?

MERGE INTO table_name

COPY INTO table_name

UPDATE table_name

INSERT INTO OVERWRITE table_name

INSERT IF EXISTS table_name

34. Data science team members are using a single cluster to perform data analysis, although cluster size was chosen to handle multiple users and auto-scaling was enabled, the team realized queries are still running slow, what would be the suggested fix for this?

Setup multiple clusters so each team member has their own cluster

Disable the auto-scaling feature

Use High concurrency mode instead of the standard mode

Increase the size of the driver node

35. Which of the following is true, when building a Databricks SQL dashboard?

A dashboard can only use results from one query

Only one visualization can be developed with one query result

A dashboard can only connect to one schema/Database

More than one visualization can be developed using a single query result

A dashboard can only have one refresh schedule

36. To reduce storage and compute costs, the data engineering team has been tasked with curating a series of aggregate tables leveraged by business intelligence dashboards, customer-facing applications, production machine learning models, and ad hoc analytical queries.

The data engineering team has been made aware of new requirements from a customer-facing application, which is the only downstream workload they manage entirely. As a result, an aggregate table used by numerous teams across the organization will need to have a number of fields renamed, and additional fields will also be added.

Which of the solutions addresses the situation while minimally interrupting other teams in the organization without increasing the number of tables that need to be managed?

Send all users notice that the schema for the table will be changing; include in the communication the logic necessary to revert the new table schema to match historic queries.

Configure a new table with all the requisite fields and new names and use this as the source for the customer-facing application; create a view that maintains the original data schema and table name by aliasing select fields from the new table.

Create a new table with the required schema and new fields and use Delta Lake's deep clone functionality to sync up changes committed to one table to the corresponding table.

Replace the current table definition with a logical view defined with the query logic currently writing the aggregate table; create a new table to power the customer-facing application.

Add a table comment warning all users that the table schema and field names will be changing on a given date; overwrite the table in place to the specifications of the customer-facing application.

37. Which of the following describes how Databricks Repos can help facilitate CI/CD workflows on the Databricks Lakehouse Platform?

Databricks Repos can facilitate the pull request, review, and approval process before merging branches

Databricks Repos can merge changes from a secondary Git branch into a main Git branch

Databricks Repos can be used to design, develop, and trigger Git automation pipelines

Databricks Repos can store the single-source-of-truth Git repository

Databricks Repos can commit or push code changes to trigger a CI/CD process

38. You are asked to setup an AUTO LOADER to process the incoming data, this data arrives in JSON format and get dropped into cloud object storage and you are required to process the data as soon as it arrives in cloud storage, which of the following statements is correct

AUTO LOADER is native to DELTA lake it cannot support external cloud object storage

AUTO LOADER has to be triggered from an external process when the file arrives in the cloud storage

AUTO LOADER needs to be converted to a Structured stream process

AUTO LOADER can only process continuous data when stored in DELTA lake

AUTO LOADER can support file notification method so it can process data as it arrives

39. You are using k-means clustering to classify heart patients for a hospital. You have chosen Patient Sex, Height, Weight, Age and Income as measures and have used 3 clusters. When you create a pair-wise plot of the clusters, you notice that there is significant overlap between the clusters .

What should you do?

Identify additional measures to add to the analysis

Remove one of the measures

Decrease the number of clusters

Increase the number of clusters

40. Which of the following data workloads will utilize a silver table as its source?

A job that enriches data by parsing its timestamps into a human-readable format

A job that queries aggregated data that already feeds into a dashboard

A job that ingests raw data from a streaming source into the Lakehouse

A job that aggregates cleaned data to create standard summary statistics

A job that cleans data by removing malformatted records

Page 5 of 11

41. You are working to set up two notebooks to run on a schedule, the second notebook is dependent on the first notebook but both notebooks need different types of compute to run in an optimal fashion, what is the best way to set up these notebooks as jobs?

Use DELTA LIVE PIPELINES instead of notebook tasks

A Job can only use single cluster, setup job for each notebook and use job dependency to link both jobs together

Each task can use different cluster, add these two notebooks as two tasks in a single job with linear dependency and modify the cluster as needed for each of the tasks

Use a single job to setup both notebooks as individual tasks, but use the cluster API to setup the second cluster before the start of second task

Use a very large cluster to run both the tasks in a single job

42. What is the main difference between the silver layer and the gold layer in medallion architecture?

Silver may contain aggregated data

Gold may contain aggregated data

Data quality checks are applied in gold

Silver is a copy of bronze data

God is a copy of silver data

43. ELSE (temp C 33 ) * 5/9 5.END

44. Which of the following array functions takes input column return unique list of values in an array?

COLLECT_LIST

COLLECT_SET

COLLECT_UNION

ARRAY_INTERSECT

ARRAY_UNION

45. where on_hand = 0

Turn on the Serverless feature for the SQL endpoint.

Increase the maximum bound of the SQL endpoint’s scaling range.

Increase the cluster size of the SQL endpoint.

Turn on the Auto Stop feature for the SQL endpoint.

Turn on the Serverless feature for the SQL endpoint and change the Spot Instance Pol-icy to “Reliability Optimized.”

46. What is the main difference between AUTO LOADER and COPY INTO?

COPY INTO supports schema evolution.

AUTO LOADER supports schema evolution.

COPY INTO supports file notification when performing incremental loads.

AUTO LOADER supports reading data from Apache Kafka

AUTO LOADER Supports file notification when performing incremental loads.

47. The DevOps team has configured a production workload as a collection of notebooks scheduled to run daily using the Jobs UI. A new data engineering hire is onboarding to the team and has requested access to one of these notebooks to review the production logic.

What are the maximum notebook permissions that can be granted to the user without allowing accidental changes to production code or data?

Can Manage

Can Edit

No permissions

Can Read

Can Run

48. What steps need to be taken to set up a DELTA LIVE PIPELINE as a job using the workspace UI?

DELTA LIVE TABLES do not support job cluster

Select Workflows UI and Delta live tables tab, under task type select Delta live tables pipeline and select the notebook

Select Workflows UI and Delta live tables tab, under task type select Delta live tables pipeline and select the pipeline JSON file

Use Pipeline creation UI, select a new pipeline and job cluster

49. A data engineer is overwriting data in a table by deleting the table and recreating the table. Another data engineer suggests that this is inefficient and the table should simply be overwritten instead.

Which of the following reasons to overwrite the table instead of deleting and recreating the table is incorrect?

Overwriting a table is an atomic operation and will not leave the table in an unfinished state

Overwriting a table maintains the old version of the table for Time Travel

Overwriting a table is efficient because no files need to be deleted

Overwriting a table results in a clean table history for logging and audit purposes

Overwriting a table allows for concurrent queries to be completed while in progress

50. When scheduling Structured Streaming jobs for production, which configuration automatically recovers from query failures and keeps costs low?

Cluster: New Job Cluster; Retries: Unlimited; Maximum Concurrent Runs: Unlimited

Cluster: New Job Cluster; Retries: None; Maximum Concurrent Runs: 1

Cluster: Existing All-Purpose Cluster; Retries: Unlimited; Maximum Concurrent Runs: 1

Cluster: New Job Cluster; Retries: Unlimited; Maximum Concurrent Runs: 1

Cluster: Existing All-Purpose Cluster; Retries: None; Maximum Concurrent Runs: 1

Page 6 of 11

51. Delete records from the transactions Delta table where transactionDate is greater than current timestamp?

DELETE FROM transactions FORMAT DELTA where transactionDate > cur-renct_timestmap()

DELETE FROM transactions if transctionDate > current_timestamp()

DELETE FROM transactions where transactionDate > current_timestamp()

DELETE FROM transactions where transactionDate > current_timestamp() KEEP_HISTORY

DELET FROM transactions where transactionDate GE current_timestamp()

52. How are Delt tables stored?

A Directory where parquet data files are stored, a sub directory _delta_log where meta data, and the transaction log is stored as JSON files.

A Directory where parquet data files are stored, all of the meta data is stored in memory

A Directory where parquet data files are stored in Data plane, a sub directory _delta_log where meta data, history and log is stored in control pane.

A Directory where parquet data files are stored, all of the metadata is stored in parquet files

Data is stored in Data plane and Metadata and delta log are stored in control pane

53. org.apache.spark.sql.parquet OPTIONS (PATH "storage-path")

D. 1. CREATE TABLE my_table (id STRING, value STRING);

E. 1. CREATE MANAGED TABLE my_table (id STRING, value STRING);

54. You noticed a colleague is manually copying the data to the backup folder prior to running an up-date command, incase if the update command did not provide the expected outcome so he can use the backup copy to replace table, which Delta Lake feature would you recommend simplifying the process?

Use time travel feature to refer old data instead of manually copying

Use DEEP CLONE to clone the table prior to update to make a backup copy

Use SHADOW copy of the table as preferred backup choice

Cloud object storage retains previous version of the file

Cloud object storage automatically backups the data

55. query = f"select * from + schema_name +"."+table_name"

56. You are currently working with the application team to setup a SQL Endpoint point, once the team started consuming the SQL Endpoint you noticed that during peak hours as the number of concur-rent users increases you are seeing degradation in the query performance and the same queries are taking longer to run, which of the following steps can be taken to resolve the issue?

They can turn on the Serverless feature for the SQL endpoint.

They can increase the maximum bound of the SQL endpoint’s scaling range.

They can increase the cluster size(2X-Small to 4X-Large) of the SQL endpoint.

They can turn on the Auto Stop feature for the SQL endpoint.

They can turn on the Serverless feature for the SQL endpoint and change the Spot In-stance Policy from “Cost optimized” to “Reliability Optimized.”

57. The current ELT pipeline is receiving data from the operations team once a day so you had setup an AUTO LOADER process to run once a day using trigger (Once = True) and scheduled a job to run once a day, operations team recently rolled out a new feature that allows them to send data every 1 min, what changes do you need to make to AUTO LOADER to process the data every 1 min.

Convert AUTO LOADER to structured streaming

Change AUTO LOADER trigger to .trigger(ProcessingTime = "1 minute")

Setup a job cluster run the notebook once a minute

Enable stream processing

Change AUTO LOADER trigger to ("1 minute")

58. Operations team is using a centralized data quality monitoring system, a user can publish data quality metrics through a webhook, you were asked to develop a process to send messages using a webhook if there is atleast one duplicate record, which of the following approaches can be taken to integrate an alert with current data quality monitoring system

Use notebook and Jobs to use python to publish DQ metrics

Setup an alert to send an email, use python to parse email, and publish a webhook message

Setup an alert with custom template

Setup an alert with custom Webhook destination

Setup an alert with dynamic template

59. When defining external tables using formats CSV, JSON, TEXT, BINARY any query on the exter-nal tables caches the data and location for performance reasons, so within a given spark session any new files that may have arrived will not be available after the initial query.

How can we address this limitation?

UNCACHE TABLE table_name

CACHE TABLE table_name

REFRESH TABLE table_name

BROADCAST TABLE table_name

CLEAR CACH table_name

60. If you create a database sample_db with the statement CREATE DATABASE sample_db what will be the default location of the database in DBFS?

Default location, DBFS:/user/

Default location, /user/db/

Default Storage account

Statement fails “Unable to create database without location”

Default Location, dbfs:/user/hive/warehouse

Page 7 of 11

61. What is the best way to query external csv files located on DBFS Storage to inspect the data using SQL?

SELECT * FROM 'dbfs:/location/csv_files/' FORMAT = 'CSV'

SELECT CS

* from 'dbfs:/location/csv_files/'

SELECT * FROM CS

'dbfs:/location/csv_files/'

You can not query external files directly, us COPY INTO to load the data into a table first

SELECT * FROM 'dbfs:/location/csv_files/' USING CSV

62. Which of the following is a correct statement on how the data is organized in the storage when when managing a DELTA table?

All of the data is broken down into one or many parquet files, log files are broken down into one or many JSON files, and each transaction creates a new data file(s) and log file. (Correct)

All of the data and log are stored in a single parquet file

All of the data is broken down into one or many parquet files, but the log file is stored as a single json file, and every transaction creates a new data file(s) and log file gets appended.

All of the data is broken down into one or many parquet files, log file is removed once the transaction is committed.

All of the data is stored into one parquet file, log files are broken down into one or many json files.

63. SELECT * FROM CUSTOMERS_2020

64. Which of the following statement is true about Databricks repos?

You can approve the pull request if you are the owner of Databricks repos

A workspace can only have one instance of git integration

Databricks Repos and Notebook versioning are the same features

You cannot create a new branch in Databricks repos

Databricks repos allow you to comment and commit code changes and push them to a remote branch

65. Which Python variable contains a list of directories to be searched when trying to locate required modules?

importlib.resource_path

sys.path

os.path

pypi.path

pylib.source

66. You have configured AUTO LOADER to process incoming IOT data from cloud object storage every 15 mins, recently a change was made to the notebook code to update the processing logic but the team later realized that the notebook was failing for the last 24 hours, what steps team needs to take to reprocess the data that was not loaded after the notebook was corrected?

Move the files that were not processed to another location and manually copy the files into the ingestion path to reprocess them

Enable back_fill = TRUE to reprocess the data

Delete the checkpoint folder and run the autoloader again

Autoloader automatically re-processes data that was not loaded

Manually re-load the data

67. A data engineer has set up two Jobs that each run nightly. The first Job starts at 12:00 AM, and it usually completes in about 20 minutes. The second Job depends on the first Job, and it starts at 12:30 AM. Sometimes, the second Job fails when the first Job does not complete by 12:30 AM.

Which of the following approaches can the data engineer use to avoid this problem?

They can set up a retry policy on the first Job to help it run more quickly

They can use cluster pools to help the Jobs run more efficiently

They can limit the size of the output in the second Job so that it will not fail as easily

They can set up the data to stream from the first Job to the second Job

They can utilize multiple tasks in a single job with a linear dependency

68. You had worked with the Data analysts team to set up a SQL Endpoint(SQL warehouse) point so they can easily query and analyze data in the gold layer, but once they started consuming the SQL Endpoint(SQL warehouse) you noticed that during the peak hours as the number of users increase you are seeing queries taking longer to finish, which of the following steps can be taken to resolve the issue?

*Please note Databricks recently renamed SQL endpoint to SQL warehouse.

They can turn on the Serverless feature for the SQL endpoint(SQL warehouse).

They can increase the maximum bound of the SQL endpoint(SQL warehouse) ’s scaling range.

They can increase the cluster size from 2X-Small to 4X-Large of the SQL end-point(SQL warehouse) .

They can turn on the Auto Stop feature for the SQL endpoint(SQL warehouse) .

They can turn on the Serverless feature for the SQL endpoint(SQL warehouse) and change the Spot Instance Policy from “Cost optimized” to “Reliability Optimized.”

69. Which of the following approaches can the data engineer use to obtain a version-controllable con-figuration of the Job’s schedule and configuration?

They can link the Job to notebooks that are a part of a Databricks Repo.

They can submit the Job once on a Job cluster.

They can download the JSON equivalent of the job from the Job’s page.

They can submit the Job once on an all-purpose cluster.

They can download the XML description of the Job from the Job’s page

70. Unity catalog simplifies managing multiple workspaces, by storing and managing permissions and ACL at _______ level

Workspace

Account (Correct)

Storage

Data pane

Control pane

Page 8 of 11

71. Your colleague was walking you through how a job was setup, but you noticed a warning message that said, “Jobs running on all-purpose cluster are considered all purpose compute", the colleague was not sure why he was getting the warning message, how do you best explain this warning mes-sage?

All-purpose clusters cannot be used for Job clusters, due to performance issues.

All-purpose clusters take longer to start the cluster vs a job cluster

All-purpose clusters are less expensive than the job clusters

All-purpose clusters are more expensive than the job clusters

All-purpose cluster provide interactive messages that can not be viewed in a job

72. FROM carts GROUP BY cartId

Expected result:

cartId items

1 [1,100,200,300,250]

FLATTEN, COLLECT_UNION

ARRAY_UNION, FLATTEN

ARRAY_UNION, ARRAY_DISTINT

ARRAY_UNION, COLLECT_SET

ARRAY_DISTINCT, ARRAY_UNION

73. schema_name = "bronze"

spark.sql(f"SELECT * FROM f{schema_name.table_name}")

spark.sql(f"SELECT * FROM {schem_name.table_name}")

spark.sql(f"SELECT * FROM ${schema_name}.${table_name}")

spark.sql(f"SELECT * FROM {schema_name}.{table_name}")

spark.sql("SELECT * FROM schema_name.table_name")

74. Which of the following developer operations in CI/CD flow can be implemented in Databricks Re-pos?

Delete branch

Trigger Databricks CICD pipeline

Commit and push code

Create a pull request

Approve the pull request

75. You are looking to process the data based on two variables, one to check if the department is supply chain and second to check if process flag is set to True

if department = “supply chain” & process:

if department == “supply chain” && process:

if department == “supply chain” & process == TRUE:

if department == “supply chain” & if process == TRUE:

if department == "supply chain" and process:

76. as total_sales from sales

77. You are currently working with the second team and both teams are looking to modify the same notebook, you noticed that the second member is copying the notebooks to the personal folder to edit and replace the collaboration notebook, which notebook feature do you recommend to make the process easier to collaborate.

Databricks notebooks should be copied to a local machine and setup source control locally to version the notebooks

Databricks notebooks support automatic change tracking and versioning

Databricks Notebooks support real-time coauthoring on a single notebook

Databricks notebooks can be exported into dbc archive files and stored in data lake

Databricks notebook can be exported as HTML and imported at a later time

78. A newly joined team member John Smith in the Marketing team who currently does not have any access to the data requires read access to customers table, which of the following statements can be used to grant access.

GRANT SELECT, USAGE TO [email protected] ON TABLE customers

GRANT READ, USAGE TO [email protected] ON TABLE customers

GRANT SELECT, USAGE ON TABLE customers TO [email protected]

GRANT READ, USAGE ON TABLE customers TO [email protected]

GRANT READ, USAGE ON customers TO [email protected]

79. A particular job seems to be performing slower and slower over time, the team thinks this started to happen when a recent production change was implemented, you were asked to take look at the job history and see if we can identify trends and root cause, where in the workspace UI can you perform this analysis?

Under jobs UI select the job you are interested, under runs we can see current active runs and last 60 days historical run

Under jobs UI select the job cluster, under spark UI select the application job logs, then you can access last 60 day historical runs

Under Workspace logs, select job logs and select the job you want to monitor to view the last 60 day historical runs

Under Compute UI, select Job cluster and select the job cluster to see last 60 day his-torical runs

Historical job runs can only be accessed by REST API

80. A dataset has been defined using Delta Live Tables and includes an expectations clause:

CON-STRAINT valid_timestamp EXPECT (timestamp > '2020-01-01')

What is the expected behavior when a batch of data containing data that violates these constraints is processed?

Records that violate the expectation are added to the target dataset and recorded as invalid in the event log.

Records that violate the expectation are dropped from the target dataset and recorded as invalid in the event log.

Records that violate the expectation cause the job to fail.

Records that violate the expectation are added to the target dataset and flagged as in-valid in a field added to the target dataset.

Records that violate the expectation are dropped from the target dataset and loaded into a quarantine table.

Page 9 of 11

81. A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to complete a task as roughly the same, but the max duration for a task to be roughly 100 times as long as the minimum.

Which situation is causing increased duration of the overall job?

Task queueing resulting from improper thread pool assignment.

Spill resulting from attached volume storage being too small.

Network latency due to some cluster nodes being in different regions from the source data

Skew caused by more data being assigned to a subset of spark-partitions.

Credential validation errors while pulling data from an external system.

82. A DELTA LIVE TABLE pipelines can be scheduled to run in two different modes, what are these two different modes?

Triggered, Incremental

Once, Continuous

Triggered, Continuous

Once, Incremental

Continuous, Incremental

83. A table is registered with the following code:

Both users and orders are Delta Lake tables.

Which statement describes the results of querying recent_orders?

All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query finishes.

All logic will execute when the table is defined and store the result of joining tables to the DBFS; this stored data will be returned when the table is queried.

Results will be computed and cached when the table is defined; these cached results will incrementally update as new records are inserted into source tables.

All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query began.

The versions of each source table will be stored in the table transaction log; query results will be saved to DBFS with each query.

84. The data engineering team is looking to add a new column to the table, but the QA team would like to test the change before implementing in production, which of the below options allow you to quickly copy the table from Prod to the QA environment, modify and run the tests?

DEEP CLONE

SHADOW CLONE

ZERO COPY CLONE

SHALLOW CLONE

METADATA CLONE

85. The data engineering team has configured a Databricks SQL query and alert to monitor the values in a Delta Lake table. The recent_sensor_recordings table contains an identifying sensor_id alongside the timestamp and temperature for the most recent 5 minutes of recordings.

The below query is used to create the alert:

The query is set to refresh each minute and always completes in less than 10 seconds. The alert is set to trigger when mean (temperature) > 120. Notifications are triggered to be sent at most every 1 minute.

If this alert raises notifications for 3 consecutive minutes and then stops, which statement must be true?

The total average temperature across all sensors exceeded 120 on three consecutive executions of the query

The recent_sensor_recordings table was unresponsive for three consecutive runs of the query

The source query failed to update properly for three consecutive minutes and then restarted

The maximum temperature recording for at least one sensor exceeded 120 on three consecutive executions of the query

The average temperature recordings for at least one sensor exceeded 120 on three consecutive executions of the query

86. The Delta Live Table Pipeline is configured to run in Production mode using the continuous Pipe-line Mode.

What is the expected outcome after clicking Start to update the pipeline?

All datasets will be updated once and the pipeline will shut down. The compute resources will be terminated

All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will be deployed for the update and terminated when the pipeline is stopped

All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will persist after the pipeline is stopped to allow for additional testing

All datasets will be updated once and the pipeline will shut down. The compute resources will persist to allow for additional testing

All datasets will be updated continuously and the pipeline will not shut down. The compute resources will persist with the pipeline (Correct)

87. Which of the following is not a privilege in the Unity catalog?

SELECT

MODIFY

DELETE

CREATE TABLE

EXECUTE

88. You are looking to process the data based on two variables, one to check if the department is supply chain or check if process flag is set to True

if department = “supply chain” | process:

if department == “supply chain” or process = TRUE:

if department == “supply chain” | process == TRUE:

if department == “supply chain” | if process == TRUE:

if department == “supply chain” or process:

89. You are noticing job cluster is taking 6 to 8 mins to start which is delaying your job to finish on time, what steps you can take to reduce the amount of time cluster startup time

Setup a second job ahead of first job to start the cluster, so the cluster is ready with re-sources when the job starts

Use All purpose cluster instead to reduce cluster start up time

Reduce the size of the cluster, smaller the cluster size shorter it takes to start the cluster

Use cluster pools to reduce the startup time of the jobs

Use SQL endpoints to reduce the startup time

90. Direct query on external files limited options, create external tables for CSV files with header and pipe delimited CSV files, fill in the blanks to complete the create table statement

CREATE TABLE sales (id int, unitsSold int, price FLOAT, items STRING)

________________

LOCATION “dbfs:/mnt/sales/*.csv”

FORMAT CSV OPTIONS (“true”,”|”)

USING CSV TYPE (“true”,”|”)

USING CSV OPTIONS (header =“true”, delimiter = ”|”) (Correct)

FORMAT CSV FORMAT TYPE (header =“true”, delimiter = ”|”)

FORMAT CSV TYPE (header =“true”, delimiter = ”|”)

Page 10 of 11

91. Which of the following scenarios is the best fit for the AUTO LOADER solution?

Efficiently process new data incrementally from cloud object storage

Incrementally process new streaming data from Apache Kafa into delta lake

Incrementally process new data from relational databases like MySQL

Efficiently copy data from data lake location to another data lake location

Efficiently move data incrementally from one delta table to another delta table

92. The data governance team is reviewing code used for deleting records for compliance with GDPR. They note the following logic is used to delete records from the Delta Lake table named users.

Assuming that user_id is a unique identifying key and that delete_requests contains all users that have requested deletion, which statement describes whether successfully executing the above logic guarantees that the records to be deleted are no longer accessible and why?

Yes; Delta Lake ACID guarantees provide assurance that the DELETE command succeeded fully and permanently purged these records.

No; the Delta cache may return records from previous versions of the table until the cluster is restarted.

Yes; the Delta cache immediately updates to reflect the latest data files recorded to disk.

No; the Delta Lake DELETE command only provides ACID guarantees when combined with the MERGE INTO command.

No; files containing deleted records may still be accessible with time travel until a VACUUM command is used to remove invalidated data files.

93. How to determine if a table is a managed table vs external table?

Run IS_MANAGED(‘table_name’) function

All external tables are stored in data lake, managed tables are stored in DELTA lake

All managed tables are stored in unity catalog

Run SQL command DESCRIBE EXTENDED table_name and check type

Run SQL command SHOW TABLES to see the type of the table

94. Once a cluster is deleted, below additional actions need to performed by the administrator

Remove virtual machines but storage and networking are automatically dropped

Drop storage disks but Virtual machines and networking are automatically dropped

Remove networking but Virtual machines and storage disks are automatically dropped

Remove logs

No action needs to be performed. All resources are automatically removed.

95. What are the different ways you can schedule a job in Databricks workspace?

Continuous, Incremental

On-Demand runs, File notification from Cloud object storage

Cron, On Demand runs

Cron, File notification from Cloud object storage

Once, Continuous

96. A newly joined team member John Smith in the Marketing team currently has access read access to sales tables but does not have access to update the table, which of the following commands help you accomplish this?

GRANT UPDATE ON TABLE table_name TO [email protected]

GRANT USAGE ON TABLE table_name TO [email protected]

GRANT MODIFY ON TABLE table_name TO [email protected]

GRANT UPDATE TO TABLE table_name ON [email protected]

GRANT MODIFY TO TABLE table_name ON [email protected]

97. In which phase of the data analytics lifecycle do Data Scientists spend the most time in a project?

Discovery

Data Preparation

Model Building

Communicate Results

98. Which of the statements are incorrect when choosing between lakehouse and Datawarehouse?

Lakehouse can have special indexes and caching which are optimized for Machine learning

Lakehouse cannot serve low query latency with high reliability for BI workloads, only suitable for batch workloads.

Lakehouse can be accessed through various API’s including but not limited to Python/R/SQL

Traditional Data warehouses have storage and compute are coupled.

Lakehouse uses standard data formats like Parquet.

99. Which of the following is the correct statement for a session scoped temporary view?

Temporary views are lost once the notebook is detached and re-attached

Temporary views stored in memory

Temporary views can be still accessed even if the notebook is detached and attached

Temporary views can be still accessed even if cluster is restarted

Temporary views are created in local_temp database

100. What statement is true regarding the retention of job run history?

It is retained until you export or delete job run logs

It is retained for 30 days, during which time you can deliver job run logs to DBFS or S3

It is retained for 60 days, during which you can export notebook run results to HTML

It is retained for 60 days, after which logs are archived

It is retained for 90 days or until the run-id is re-used through custom run configuration

Page 11 of 11

101. You are currently working to ingest millions of files that get uploaded to the cloud object storage for consumption, and you are asked to build a process to ingest this data, the schema of the file is expected to change over time, and the ingestion process should be able to handle these changes automatically.

Which of the following method can be used to ingest the data incrementally?

AUTO APPEND

AUTO LOADER

COPY INTO

Structured Streaming

Checkpoint

102. An upstream system is emitting change data capture (CDC) logs that are being written to a cloud object storage directory. Each record in the log indicates the change type (insert, update, or delete) and the values for each field after the change. The source table has a primary key identified by the field pk_id.

For auditing purposes, the data governance team wishes to maintain a full record of all values that have ever been valid in the source system. For analytical purposes, only the most recent value for each record needs to be recorded. The Databricks job to ingest these records occurs once per hour, but each individual record may have changed multiple times over the course of an hour.

Which solution meets these requirements?

Create a separate history table for each pk_id resolve the current state of the table by running a union all filtering the history tables for the most recent state.

Use MERGE INTO to insert, update, or delete the most recent entry for each pk_id into a bronze table, then propagate all changes throughout the system.

Iterate through an ordered set of changes to the table, applying each in turn; rely on Delta Lake's versioning ability to create an audit log.

Use Delta Lake's change data feed to automatically process CDC data from an external system, propagating all changes to all dependent tables in the Lakehouse.

Ingest all log information into a bronze table; use MERGE INTO to insert, update, or delete the most recent entry for each pk_id into a silver table to recreate the current table state.

103. The data engineering team noticed that one of the job normally finishes in 15 mins but gets stuck randomly when reading remote databases due to a network packet drop, which of the following steps can be used to improve the stability of the job?

Use Databrick REST API to monitor long running jobs and issue a kill command

Use Jobs runs, active runs UI section to monitor and kill long running job

Modify the task, to include a timeout to kill the job if it runs more than 15 mins.

Use Spark job time out setting in the Spark UI

Use Cluster timeout setting in the Job cluster UI

104. You have noticed the Data scientist team is using the notebook versioning feature with git integra-tion, you have recommended them to switch to using Databricks Repos, which of the below reasons could be the reason the why the team needs to switch to Databricks Repos.

Databricks Repos allows multiple users to make changes

Databricks Repos allows merge and conflict resolution

Databricks Repos has a built-in version control system

Databricks Repos automatically saves changes

Databricks Repos allow you to add comments and select the changes you want to commit.

105. Which of the following command can be used to drop a managed delta table and the underlying files in the storage?

DROP TABLE table_name CASCADE

DROP TABLE table_name

Use DROP TABLE table_name command and manually delete files using com-mand dbutils.fs.rm("/path",True)

DROP TABLE table_name INCLUDE_FILES

DROP TABLE table and run VACUUM command

106. An upstream system has been configured to pass the date for a given batch of data to the Databricks Jobs API as a parameter. The notebook to be scheduled will use this parameter to load data with the following code: df = spark.read.format("parquet").load(f"/mnt/source/(date)")

Which code block should be used to create the date Python variable used in the above code block?

date = spark.conf.get("date")

input_dict = input() date= input_dict["date"]

import sys date = sys.argv[1]

date = dbutils.notebooks.getParam("date")

dbutils.widgets.text("date", "null") date = dbutils.widgets.get("date")

107. The Databricks workspace administrator has configured interactive clusters for each of the data engineering groups. To control costs, clusters are set to terminate after 30 minutes of inactivity. Each user should be able to execute workloads against their assigned clusters at any time of the day.

Assuming users have been added to a workspace but not granted any permissions, which of the following describes the minimal permissions a user would need to start and attach to an already configured cluster.

"Can Manage" privileges on the required cluster

Workspace Admin privileges, cluster creation allowed, "Can Attach To" privileges on the required cluster

Cluster creation allowed, "Can Attach To" privileges on the required cluster

"Can Restart" privileges on the required cluster

Cluster creation allowed, "Can Restart" privileges on the required cluster

108. A junior data engineer has configured a workload that posts the following JSON to the Databricks REST API endpoint 2.0/jobs/create.

Assuming that all configurations and referenced resources are available, which statement describes the result of executing this workload three times?

Three new jobs named "Ingest new data" will be defined in the workspace, and they will each run once daily.

The logic defined in the referenced notebook will be executed three times on new clusters with the configurations of the provided cluster I

Three new jobs named "Ingest new data" will be defined in the workspace, but no jobs will be executed.

One new job named "Ingest new data" will be defined in the workspace, but it will not be executed.

The logic defined in the referenced notebook will be executed three times on the referenced existing all purpose cluster.

109. A data architect is designing a data model that works for both video-based machine learning work-loads and highly audited batch ETL/ELT workloads.

Which of the following describes how using a data lakehouse can help the data architect meet the needs of both workloads?

A data lakehouse requires very little data modeling

A data lakehouse combines compute and storage for simple governance

A data lakehouse fully exists in the cloud

A data lakehouse stores unstructured data and is ACID-compliant

A data lakehouse provides autoscaling for compute clusters

110. A nightly job ingests data into a Delta Lake table using the following code:

The next step in the pipeline requires a function that returns an object that can be used to manipulate new records that have not yet been processed to the next table in the pipeline.

Which code snippet completes this function definition?

def new_records():

A) return spark.readStream.table("bronze")

B) return spark.readStream.load("bronze")

C)

D)return spark.read.option("readChangeFeed", "true").table ("bronze")

E)

Option A

Option B

Option C

Option D

TAGS:

Databricks Certified Data Engineer Professional, Databricks Certified Data Engineer Professional exam dumps

Notify of

Label

Name*

Email*

Website

Label

Name*

Email*

Website

0 Comments

Oldest

Newest Most Voted

Inline Feedbacks

View all comments

Get Databricks Certified Data Engineer Professional Dumps Full Version

Q&As: 337
Versions: PDF and Software Download Now

About Dumpsinfo

Dumpsinfo is a good platform providing the latest exam information and dumps questions for all IT certification exams. You can study all the latest exam dumps questions online.

[email protected]

Mon - Sat 9:00am - 6:00pm

Use Databricks Certified Data Engineer Professional Dumps to Obtain Databricks Certification

Related

Posts

Databricks Generative AI Engineer Associate Dumps Questions – Effective Way to Get Certified

Pass Databricks Machine Learning Associate Exam to Get Databricks Certification

Good Databricks Certified Professional Data Engineer Exam Dumps Save Your Preparation Time

Databricks Certified Data Analyst Associate Dumps Questions – Effective Way to Get Certified

About Us

EMAIL

Services

Opening Hours