Feedback Loops: Optimizing Image Classifiers in 3 Steps

Salesforce Einstein Vision gives you a set of powerful API's that can be used to bring image recognition capabilities into your CRM application. Image recognition can be used for Visual search, Brand detection, Product identification and many more. With Salesforce Einstein Vision, you can leverage pre-trained image classifiers, or train your own custom classifiers to solve a vast array of image-recognition use cases, empowering end users across sales, service, and marketing to be smarter and more predictive.

What is Feedback loop?

A feedback loop refers to the process by which an AI model’s predicted outputs are reused to train new versions of the model. Salesforce Einstein Vision Platform provides set of powerful APIs for  adding feedback loop which improves the image classifier's prediction capabilities by using the model's misclassified images to train new versions of the model.

Why we need Feedback loop?

In today's world every industry uses feedback loops to streamline their operations. In business, the practice of taking customer feedback (the output of a product or service) and using it to improve future processes is best example and commonly used feedback loop.

Feedback loops ensure that AI results do not stagnate. This also has a significant advantage as the data used to train new versions of the model is of the same real-world distribution that the customer cares about predicting over. Without it, AI will choose the path of least resistance, even when that path is wrong, causing its performance to deteriorate. By incorporating a feedback, you can reinforce your model's training and keep improving them over time.

Completing the Feedback loops with Einstein Vision
In many applications developer train their AI models with sets containing large number of labeled image data. Over the time, the prediction accuracy might decline for some set of images.
Luckily, Einstein Vision APIs provides a way to limit this issue and keep your prediction results accurate everytime.

Feedback loop with Einstein Vision APIs can be used in 3 simple steps

1. Add a misclassified image to a dataset with the correct label.
2. Get a list of images that were added as feedback to a dataset.
3. Create a model or update an existing model using feedback images.

Let’s look at an example. Let’s say you have an image classification model that classifies beaches and mountains. You send in an image, LakeSideMontain.jpg, to the model to get back a prediction.

The model returns a response that contains a high probability that the image is a beach (in the Beaches class). However, you expect a response that contains a high probability that the image is a mountain (in the Mountains class). This means that the image was misclassified.

LakeSideMontain.jpg

This misclassified image can be used to provide feedback to the model in order to increase accuracy for the future predictions.

How it's done?
Before we start, if this is the first time you are going use and explore Einstein platform APIs, I would recommend you to go through the Trailhead modules to gain basic knowledge on Einstein. Once you have some basic knowledge you can start with Einstein Vision and Language Model Builder app, created by Salesforce Labs, is available for free on AppExchange. This app is super helpful for quickly uploading dataset, training models and predicting the results. Follow the instructions in the Quip doc.

Once you are familiar with the basic Einstein Vision APIs, you can refine your model by using below feedback APIs

Step 1:
The first step is to add the misclassified images to the dataset along with their correct labels.

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=<MODELID>" -F "data=@c:\data\LakeSideMontain.jpg" -F "expectedLabel=Mountains" https://api.einstein.ai/v2/vision/feedback

Step 2:
After we add feedback examples to a dataset, we can query the dataset and return just those examples that were added from the feedback API call.

curl -X GET -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/datasets/<DATASET_ID>/examples?source=feedback

Note: Additional source parameter is used to query only feedback images 

Step 3:
Once we retrieve and validate the feedback images we have 2 options to train our model

• Create new Model
• Update existing Model

Updating existing model approach is useful in production environment where it will train the existing model with feedback examples and preserve our model ID.
Additional request parameter  trainParams={"withFeedback" : true} is used with retrain endpoint.

curl -X GET -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=<MODELID>" -F "trainParams={\"withFeedback\" : true}" https://api.einstein.ai/v2/vision/retrain

Additional Resources

Documentation: Einstein Platform Services Developer Guide
Other Resource: Get Started With Building Einstein Vision and Language Models
Trailhead Project: Quick Start: Einstein Image Classification