You have written a notebook to generate a summary data set for reporting, Notebook was scheduled using the job cluster, but you realized it takes 8 minutes to start the cluster, what feature can be used to start the cluster in a timely fashion so your job can run immediatley?

You have written a notebook to generate a summary data set for reporting, Notebook was scheduled using the job cluster, but you realized it takes 8 minutes to start the cluster, what feature can be used to start the cluster in a timely fashion so your job can run immediatley?
A . Setup an additional job to run ahead of the actual job so the cluster is running second job starts
B. Use the Databricks cluster pools feature to reduce the startup time
C. Use Databricks Premium edition instead of Databricks standard edition
D. Pin the cluster in the cluster UI page so it is always available to the jobs
E. Disable auto termination so the cluster is always running

View Answer

Answer: B

Explanation:

Cluster pools allow us to reserve VM’s ahead of time, when a new job cluster is created VM are grabbed from the pool. Note: when the VM’s are waiting to be used by the cluster only cost incurred is Azure. Databricks run time cost is only billed once VM is allocated to a cluster.

Here is a demo of how to setup a pool and follow some best practices,

Graphical user

interface, text

Description automatically generated