What would you recommend for reducing the overall cost of this approach?

You noticed that a team member started using an all-purpose cluster to develop a notebook and used the same all-purpose cluster to set up a job that can run every 30 mins so they can update un-derlying tables which are used in a dashboard.

What would you recommend for reducing the overall cost of this approach?
A . Reduce the size of the cluster
B . Reduce the number of nodes and enable auto scale
C . Enable auto termination after 30 mins
D . Change the cluster all-purpose to job cluster when scheduling the job
E . Change the cluster mode from all-purpose to single-mode

View Answer

Answer: D

Explanation:

While using an all-purpose cluster is ok during development but anytime you don’t need to interact with a notebook, especially for a scheduled job it is less expensive to use a job cluster. Using an all-purpose cluster can be twice as expensive as a job cluster.

Please note: The compute cost you pay the cloud provider for the same cluster type and size be-tween an all-purpose cluster and job cluster is the same the only difference is the DBU cost.

The total cost of cluster = Total cost of VM compute (Azure or AWS or GCP) + Cost per DBU

The per DBU cost varies between all-purpose and Job Cluster

Here is the recent cost estimate from AWS between Jobs Cluster and all-purpose Cluster, for jobs compute its $0.15 cents per DBU v$0.55 cents per DBU for all-purpose

Graphical user

interface

Description automatically generated

How do I check how much the DBU cost for my cluster?

When you click on an exister cluster or when you look at the cluster details you will see this in the top right corner

Graphical user

interface, text, application, email

Description automatically generated