Which of the following approaches could be used by the data engineering team to complete this task?

A data analyst has a series of queries in a SQL program. The data analyst wants this program to run every day. They only want the final query in the program to run on Sundays. They ask for help from the data engineering team to complete this task.

Which of the following approaches could be used by the data engineering team to complete this task?
A . They could submit a feature request with Databricks to add this functionality.
B . They could wrap the queries using PySpark and use Python’s control flow system to determine when to run the final query.
C . They could only run the entire program on Sundays.
D . They could automatically restrict access to the source table in the final query so that it is only accessible on Sundays.
E . They could redesign the data model to separate the data used in the final query into a new table.

View Answer

Answer: B

Explanation:

This approach would allow the data engineering team to use the existing SQL program and add some logic to control the execution of the final query based on the day of the week. They could use the datetime module in Python to get the current date and check if it is a Sunday. If so, they could run the final query, otherwise they could skip it. This way, they could schedule the program to run every day without changing the data model or the source table.

Reference: PySpark SQL Module, Python datetime Module, Databricks Jobs