What is the expected behavior when a batch of data containing data that violates these constraints is processed?

A dataset has been defined using Delta Live Tables and includes an expectations clause: CON-STRAINT valid_timestamp EXPECT (timestamp > ‘2020-01-01’) ON VIOLATION FAIL

What is the expected behavior when a batch of data containing data that violates these constraints is processed?
A . Records that violate the expectation are added to the target dataset and recorded as invalid in the event log.
B. Records that violate the expectation are dropped from the target dataset and recorded as invalid in the event log.
C. Records that violate the expectation cause the job to fail
D. Records that violate the expectation are added to the target dataset and flagged as in-valid in a field added to the target dataset.
E. Records that violate the expectation are dropped from the target dataset and loaded into a quarantine table.

View Answer

Answer: C

Explanation:

The answer is Records that violate the expectation cause the job to fail.

Delta live tables support three types of expectations to fix bad data in DLT pipelines Review below example code to examine these expectations,

Diagram

Description automatically generated with medium confidence

Invalid records:

Use the expect operator when you want to keep records that violate the expectation.

Records that violate the expectation are added to the target dataset along with valid records:

SQL

CONSTRAINT valid_timestamp EXPECT (timestamp > ‘2020-01-01’)

Drop invalid records:

Use the expect or drop operator to prevent the processing of invalid records. Records that violate the expectation are dropped from the target dataset: SQL

CONSTRAINT valid_timestamp EXPECT (timestamp > ‘2020-01-01’) ON VIOLATION DROP ROW

Fail on invalid records:

When invalid records are unacceptable, use the expect or fail operator to halt execution immediately when a record fails validation.

If the operation is a table update, the system atomically rolls back the transaction:

SQL

CONSTRAINT valid_timestamp EXPECT (timestamp > ‘2020-01-01’) ON VIOLATION FAIL UP-DATE