How Data Engineer can do Monitoring of Files which are Staged Internally during Continuous data pipelines loading process? [Select all that apply]

How Data Engineer can do Monitoring of Files which are Staged Internally during Continuous data pipelines loading process? [Select all that apply]
A . She Can Monitor the files using Metadata maintained by Snowflake i.e. file-name,last_modified date etc.
B . Snowflake retains historical data for COPY INTO commands executed within the previous 14 days.
C . She can Monitor the status of each COPY INTO <table> command on the History tab page of the classic web interface.
D . She can use the DATA_LOAD_HISTORY Information Schema view to retrieve the history of data loaded into tables using the COPY INTO command.
E . She can use the DATA_VALIDATE function to validate the data files She have loaded and can retrieve any errors encountered during the load.

View Answer

Answer: A, B, C

Explanation:

Monitoring Files Staged Internally

Snowflake maintains detailed metadata for each file uploaded into internal stage (for users, tables, and stages), including:

・ File name

・ File size (compressed, if compression was specified during upload)

・ LAST_MODIFIED date, i.e. the timestamp when the data file was initially staged or when it was last modified, whichever is later

In addition, Snowflake retains historical data for COPY INTO commands executed within the previous 14 days. The metadata can be used to monitor and manage the loading process, including deleting files after upload completes:

・ Use the LIST command to view the status of data files that have been staged.

・ Monitor the status of each COPY INTO <table> command on the History tab page of the classic web interface.

・ Use the VALIDATE function to validate the data files you’ve loaded and retrieve any errors encountered during the load.

・ Use the LOAD_HISTORY Information Schema view to retrieve the history of data loaded into tables using the COPY INTO command.