Which of the following would the Specialist do to integrate the Spark application with SageMaker?

A Machine Learning Specialist is using Apache Spark for pre-processing training data As part of the Spark pipeline, the Specialist wants to use Amazon SageMaker for training a model and hosting it.

Which of the following would the Specialist do to integrate the Spark application with SageMaker? (Select THREE)
A . Download the AWS SDK for the Spark environment
B . Install the SageMaker Spark library in the Spark environment.
C . Use the appropriate estimator from the SageMaker Spark Library to train a model.
D . Compress the training data into a ZIP file and upload it to a pre-defined Amazon S3 bucket.
E . Use the sageMakerModel. transform method to get inferences from the model hosted in SageMaker
F . Convert the DataFrame object to a CSV file, and use the CSV file as input for obtaining inferences from SageMaker.

View Answer

Answer: B, C, E

Explanation:

The SageMaker Spark library is a library that enables Apache Spark applications to integrate with Amazon SageMaker for training and hosting machine learning models.

The library provides several features, such as:

Estimators: Classes that allow Spark users to train Amazon SageMaker models and host them on Amazon SageMaker endpoints using the Spark MLlib Pipelines API. The library supports various built-in algorithms, such as linear learner, XGBoost, K-means, etc., as well as custom algorithms using Docker containers.

Model classes: Classes that wrap Amazon SageMaker models in a Spark MLlib Model abstraction. This allows Spark users to use Amazon SageMaker endpoints for inference within Spark applications. Data sources: Classes that allow Spark users to read data from Amazon S3 using the Spark Data Sources API. The library supports various data formats, such as CSV, LibSVM, RecordIO, etc.

To integrate the Spark application with SageMaker, the Machine Learning Specialist should do the following:

Install the SageMaker Spark library in the Spark environment. This can be done by using Maven, pip, or downloading the JAR file from GitHub.

Use the appropriate estimator from the SageMaker Spark Library to train a model.

For example, to train a linear learner model, the Specialist can use the following code:

Use the sageMakerModel. transform method to get inferences from the model hosted in SageMaker.

For example, to get predictions for a test DataFrame, the Specialist can use the following code:

References:

[SageMaker Spark]: A documentation page that introduces the SageMaker Spark library and its features.

[SageMaker Spark GitHub Repository]: A GitHub repository that contains the source code, examples, and installation instructions for the SageMaker Spark library.