Automated Machine Learning Architecture
AI Azure Machine Learning Clemens Siebler  

Using Automated Machine Learning for building and deploying Models as APIs

In this post we’ll look into using Azure Automated Machine Learning for deploying Machine Learning Models as APIs into production. As an example, we will be training and deploying a simple text sentiment analysis service, using the IMDB reviews dataset (subsampled to 1000 examples).

We will achieve this by building the following architecture:

Automated Machine Learning Architecture

In detail, we will perform the following steps:

  1. Use Machine Learning Services Workspace as the central orchestration pane
  2. Upload our training data to Azure Blob
  3. Deploy an Azure Machine Learning Compute cluster for running our Automated Machine Learning training job
  4. Once the job has completed, we will containerize the best model and push it to Azure Container Registry
  5. Deploy our container to Azure Container Instances

Quite a mouthful, but with a little bit of Python, this can be achieved in less than an hour. By the way, this is the first part of a series of posts, the second part is about taking the model generated in this post and deploying it to Azure Kubernetes Service. Nevertheless, let’s get started!

Automated Machine Learning setup

Firstly, we want to deploy our Machine Learning Services workspace as outlined in this guide. Once completed, we’ll fire up our preferred Jupyter environment (I am using the free Azure Notebooks) and get started with our required imports:

import logging
import os
import json

from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
from sklearn import datasets

import azureml.core
from azureml.core.experiment import Experiment
from azureml.core.workspace import Workspace
from azureml.core.dataset import Dataset
from azureml.train.automl import AutoMLConfig

from azureml.core.compute import AmlCompute
from azureml.core.compute import ComputeTarget

Next, we want connect to our Workspace and create a new experiment (we need to make sure the config.json connection file has been put into the same folder as our Jupyter notebook):

ws = Workspace.from_config()

experiment_name = 'imdb-sentiment'
project_folder = './sentiment-output/'

experiment = Experiment(ws, experiment_name)

Preparing our datastore

Let’s first have a short look at our data in imdb_sentiment.csv:

0,A very very very slow-moving aimless movie about a distressed drifting young man.
0,Not sure who was more lost - the flat characters or the audience nearly half of whom walked out.
0,Attempting artiness with black & white and clever camera angles the movie disappointed - became even more ridiculous - as the acting was poor and the plot and lines almost non-existent.
0,Very little music or anything to speak of.
1,The best scene in the movie was when Gerardo is trying to find a song that keeps running through his head.

It is important that the data is properly formatted, e.g., in our case being in a consistent <class>,<text> format.

Now that our data looks good, we’ll upload it into the default datastore that has been provisioned with our workspace. In case we need another Storage Account, we can reference it using Datastore.register_azure_blob_container(...) (full guide here).

In our case, we’ll just upload the data-imdb folder (containing a single CSV file) from the local directory to Azure Blob:

ds = ws.get_default_datastore()
ds.upload(src_dir='./data-imdb', target_path='data-imdb', overwrite=True, show_progress=True)

Next, we’ll create a dataset reference – in this case, the data will get loaded on demand from Azure Blob:

dataset = Dataset.Tabular.from_delimited_files(path=ds.path('data-imdb/imdb_sentiment.csv'))
X = dataset.keep_columns(columns=['Text'])
y = dataset.keep_columns(columns=['Sentiment'])

We can do a quick test and check if everything looks reasonable:


data is looking good

Everything looks good, so we can move on to the next steps!

Creating our Compute Cluster

We can either run our Automated Machine Learning job locally or on a remote cluster. Locally may require a beefy machine as we want to evaluate a lot of different Machine Learning pipelines (a combination of preprocessing + algorithm + hyperparameters). Hence, we’ll create our compute cluster using Azure Machine Learning compute and in this case, we’ll be using a single node Standard_F16s_v2 cluster:

amlcompute_cluster_name = "cpu-f16sv2"

# Check if this compute target already exists in the workspace
cts = ws.compute_targets
if amlcompute_cluster_name in cts and cts[amlcompute_cluster_name].type == 'AmlCompute':
    found = True
    print('Found existing compute target, will use it!')
    compute_target = cts[amlcompute_cluster_name]
    print('Creating a new compute target...')
    provisioning_config = AmlCompute.provisioning_configuration(vm_size = "Standard_F16s_v2", min_nodes = 1, max_nodes = 1,
    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, provisioning_config)

print('Waiting for cluster creation completion...')
compute_target.wait_for_completion(show_output = True, timeout_in_minutes = 20)

print('Cluster is ready!')

In the eastus region, our cluster should be ready in 1-2 minutes.

Model Training

Finally, we can configure our Conda environment for the model training and kick off our Automated Machine Learning job:

from azureml.core.runconfig import RunConfiguration
from azureml.core.conda_dependencies import CondaDependencies
import pkg_resources

conda_run_config = RunConfiguration(framework="python") = compute_target
conda_run_config.environment.docker.enabled = True

cd = CondaDependencies.create(conda_packages=['numpy','scikit-learn','py-xgboost<=0.80'],

conda_run_config.environment.python.conda_dependencies = cd

In our case, we will allow a maximum of 20 iterations, meaning we’ll test 20 different pipelines. The preprocess flag tells AutoML to perform additional pre-processing on the input data. This step is required as our input data consists of text, which Automated Machine Learning converts to vectors using word embeddings. We’ll also turn on enable_early_stopping which automatically terminates poorly performing pipelines:

automl_settings = {
    "iteration_timeout_minutes": 5,
    "iterations": 20,
    "n_cross_validations": 5,
    "primary_metric": 'AUC_weighted',
    "preprocess": True,
    "max_concurrent_iterations": 3,
    "enable_early_stopping": True,
    "verbosity": logging.INFO

automl_config = AutoMLConfig(task = 'classification',
                             debug_log = 'automl_errors.log',
                             path = project_folder,
                             X = X,
                             y = y,

remote_run = experiment.submit(automl_config, show_output = False)

As we do not have an additional testing data set available, we’ll use 5-fold cross validation. Once the job starts running, we can either monitor it in the Azure Portal, or using the little widget from azureml.widgets:

from azureml.widgets import RunDetails

Training widget for our Automated Machine Learning job

Time to sit back, relax, and wait until our job has been completed:

remote_run.wait_for_completion(show_output = False)

At this point, we will have trained a set of Machine Learning models, including data preprocessing. Our widget (and also the Azure Portal) will show the progress and accuracy of each model quite nicely:

Training success

Now it is time to pick the best performing model and deploy it to an Azure Container Instance!

Model Deployment

Firstly, we want to pick the best performing model:

best_run, fitted_model = remote_run.get_output()
print("Run:", best_run)
print("Model:", fitted_model)

Next, we can give it a short test-drive, just to make sure it is working properly:

test = pd.DataFrame(['the food was horrible', 'wow, this movie was truely great, I totally enjoyed it!'], columns = ['Text'])

model sanity check

Looks good – the first sentence is correctly classified as negative sentiment, the second one classified as positive sentiment.

Now we can register the model in our Workspace. We do this in order to have our “hand-picked” model tracked by the workspace.

description = 'AutoML Model for Sentiment'
tags = {"dataset": "imdb-sentiment"}
model = remote_run.register_model(description = description, tags = tags)


Register Model

Next, we need a scoring file that will load our model and send data to it. However, we need to make sure that we’ll replace model_name with the model id from our registration step above:

import pickle
import json
import pandas as pd
import azureml.train.automl
from sklearn.externals import joblib
from azureml.core.model import Model

def init():
    global model
    model_path = Model.get_model_path(model_name = "AutoMLxxxxxxbest")
    model = joblib.load(model_path)

def run(rawdata):
        data = json.loads(rawdata)['text']
        result = model.predict(pd.DataFrame(data, columns = ['Text']))
    except Exception as e:
        result = str(e)
        return json.dumps({"error": result})
    return json.dumps({"result": result.tolist()})

Lastly, we need to specify how our environment should look like. In this case, we’ll just use the same definition as we did during the training step and write it out to a file:

conda_env_file_name = 'automl-imdb-sentiment-env.yml'
cd.save_to_file('.', conda_env_file_name)

Now, we can build our container image containing our and the Conda environmentautoml-imdb-sentiment-env.yml file:

from azureml.core.image import Image, ContainerImage

image_config = ContainerImage.image_configuration(runtime= "python",
                                 execution_script = "",
                                 conda_file = conda_env_file_name,
                                 tags = tags,
                                 description = "IMDB Sentimemt API")

image = Image.create(name = "imdb-sentiment-automl",
                     models = [model],
                     image_config = image_config, 
                     workspace = ws)

image.wait_for_creation(show_output = True)

if image.creation_state == 'Failed':
    print("Image build log at: " + image.image_build_log_uri)

And last but not least, we can finally initiate the deployment to Azure Container Instances. In a production scenario, I would recommend to deploy to Azure Kubernetes Service, but since the methodology is very similar, we’ll skip it for here:

from azureml.core.webservice import AciWebservice
from azureml.core.webservice import Webservice

aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, 
                                               memory_gb = 1, 
                                               tags = tags, 
                                               description = description)

aci_service_name = 'automl-imdb-sentiment'
aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,
                                           image = image,
                                           name = aci_service_name,
                                           workspace = ws)

After a couple of minutes, the service should be deployed and we can finally test it!

Testing our Web Service

To make sure our service is working, we can send a request containing multiple text inputs to it:

import requests
import json

headers = {'Content-Type':'application/json'}
data = {"text": ['the food was horrible', 
                 'wow, this movie was truely great, I totally enjoyed it!',
                 'why the heck was my package not delivered in time?']}

resp =, json=data, headers=headers)
print("Prediction Results:", resp.json())

Looking at the results, everything seems reasonable:

API results


Using Azure Automated Machine Learning is not rocket science on Azure. Data can be easily uploaded and then accessed from Azure Blob and subsequently processed in a multi-node Azure Machine Learning Compute cluster for automatically generating and evaluating Machine Learning pipelines.

In order to deploy the best model, we containerized and then deployed it to Azure Container Instances. In this second post, we’re covering how we can also deploy the same model to Azure Kubernetes Service which is the recommended, production-approved way that gives us a lot more flexibility. Fore more information around Azure Machine Learning Services, you can find plenty of further examples in the official Azure Machine Learning Notebooks Repository.

If you have any questions or want to give feedback, feel free to reach out via Twitter or just comment below.

Leave A Comment