Model Handler Function FOR 18.10

This page describes the handler functions available for the 18.10 image.
For the handler functions that can be used with after the 19.04 image, please refer to here.

The model handler function is a function in the code called when executing the model. The handler function uses the following syntax structure.

def handler(input_iter, context):
    for line in input_iter:
        ...
        yield ret_line

input_iter : Input data is stored in this parameter. This parameter is usually a Python iterator.
context : Runtime metadata, etc. are stored in this parameter.

Hander Arguments 　

Input Data

Input data is passed as an iterator to the first argument of the handler function. The format of the data contained in the iterator differs for each MIME Type.

JSON

The input JSON is passed as the dict or list type to the first element of the iterator. The length of the iterator is one.

It corresponds to the following MIME Type.

application/json

Image

The input image is passed as the numpy.ndarray type to the first element of the iterator. The length of the iterator is one. the order of color of numpy.ndarray is RGB.

It corresponds to the following MIME Type.

image/jpeg
image/png

Video

Video is supported in trigger and run for now.

The input video is passed as an iterator that returns the following dict type.

{
  "image": `numpy.ndarray`,
  "pos_frames": `int`,
  "pos_msec": `int`
}

key	type	description
`image`	`numpy.ndarray`	an array object of frame image ordered by RGB
`pos_frames`	`int`	0-based index of the frame
`pos_msec`	`int`	current position of the video file in milliseconds

It corresponds to the following MIME Type.

video/avi
video/x-matroska
video/quicktime
video/mp4

Context Data

Context variable has the following properties as context information is passed as the second argument of the handler function.

Property Name	Description
datatypes	A module containing a type that can be processed as a return value on ABEJA Platform
exceptions	A module containing a type that can be handled as an exception on ABEJA Platform
http_method	Executed HTTP method
http_headers	A dict containing all HTTP request headers. Note all headers will be downcased. Some headers (including `Authorization` header) will be removed
parameters	Runtime meta information and other parameters

Handler Environment Variable

The handler function can use the following environment variables.

Environment variable name	Description
ABEJA_ORGANIZATION_ID	ID of the organization
ABEJA_MODEL_ID	ID of the model
ABEJA_MODEL_VERSION_ID	ID of the model version
ABEJA_DEPLOYMENT_ID	ID of the model deployment
ABEJA_SERVICE_ID	ID of the service
HANDLER	Path of the handler
TRAINING_JOB_DEFINITION_NAME	Training Job Definition name
TRAINING_JOB_ID	Training Job ID
ABEJA_TRAINING_RESULT_DIR	Relative path to the directory where the result of the training job is stored

In addition to the above, environment variables that can be specified when creating code version/service/trigger/run can also be used.
For more information on user-specifiable environment variables, see here.

Use of training result in the model handler

ABEJA Platform allows you to create a model version based on the result of training.

In order to use the training output in the model handler function, it must be stored in the directory specified by the ABEJA_TRAINING_RESULT_DIR environment variable at the time of training. The output of training is extracted to the directory stored in the ABEJA_TRAINING_RESULT_DIR environment variable when the model handler function is executed.

Handler return value and exception

Return value

In order to output data, we return data of Response type contained in datatypes of context data as a generator.

context.datatypes.Response([Data to return], metadata=[('Content-Type', returning MIME Type)])

The data is returned as an array to the first argument of the Response. metadata is used as a response header.

example of data to be returned and MIME Type are given below.

MIME Type	return type
application/json	`dict`or`list`
image/jpeg	`bytes`
image/png	`bytes`

Since ABEJA Platform treats application/json as the default MIME Type, if you want to use JSON as a response, do not use Response type. It is also possible to return dict or list.

Exceptions

Exception defined by ABEJA Platform is handled in the handler function.

The following exception classes are included in the exceptions property of the context data.

context.exceptions.InvalidInput
context.exceptions.ModelError

Example of implementing handler function

The following is a simple implementation example of a model handler function. You can check the operation by using the CLI run-local command.

Input JSON and return JSON

Here is an example of performing a simple calculation on the input JSON and returning the JSON response.

Get input data from the iterator input_iter as follows

When outputting JSON, model handler function should return generator which yields context.datatypes.Response type. The context.datatypes.Response type object should include JSON response data as dict or list type and application/json Content-type in the metadata key argument like below.

def handler(input_iter, context):
    for line in input_iter:
        total_without_tax = line['total_without_tax']
        total_without_tax = int(total_without_tax)
        total = total_without_tax * 1.08
        yield context.datatypes.Response([{'total': total}], metadata=[('Content-Type', 'application/json')])

If you return dict or list without using Response type for JSON response, it will return same response as above.

def handler(input_iter, context):
    for line in input_iter:
        total_without_tax = line['total_without_tax']
        total_without_tax = int(total_without_tax)
        total = total_without_tax * 1.08
        yield {'total': total}

The output of executing run-local command. The output result for request.json as input is displayed.

$ abeja model run-local --image abeja/all-cpu:18.10 --handler main:handler --input request.json --quiet
{
    "total": 108.0
}

Input an image and return an image

This is an example of returning the input image horizontally flipped.

Input iterator input_iter contains input data as JSON format. Get input data as follows.

We implement a generator that returns the flipped input image as bytes type in context.datatypes.Response type. In this example, since the output is a JPEG image, set Content-Type as image/jpeg in metadata as follows.

import cv2

def handler(input_iter, context):
    for img in input_iter:
        flipped_image = cv2.flip(img, 1)
        _, encoded_flipped_image = cv2.imencode('.jpeg', cv2.cvtColor(flipped_image, cv2.COLOR_BGR2RGB))

        yield context.datatypes.Response(
            [encoded_flipped_image.tostring()],
            metadata=[('Content-Type', 'image/jpeg')]
        )

It is the execution result of the run-local command. A flipped image of cat.jpeg which is input is output.

$ abeja model run-local --image abeja/all-cpu:18.10 --handler main:handler --input cat.jpeg --quiet > flipped_cat.jpeg
$ open flipped_cat.jpeg

Using the training result and returning JSON from the image.

We return the inference result of the model using the HDF 5 file output as the result of training by JSON.

Reads the HDF5 file from the training result extracted to ABEJA_TRAINING_RESULT_DIR and makes an inference. At this time, it is necessary to specify the same file name as the HDF5 file saved in the output directory during training.

import os
from keras.models import load_model
from PIL import Image

model = load_model(os.path.join(os.environ.get('ABEJA_TRAINING_RESULT_DIR', '.'), 'model.h5'))

def handler(input_iter, ctx):
    for img in input_iter:
        img = Image.fromarray(img)

        # PREPROCESS INPUT HERE
        ...

        result = model.predict(img)

        # CONVERT RESULT FOR RESPONSE
        ...

        yield {"result": result}

It is the execution result of the run-local command. Inference result with input cat.jpeg as input is returned.

$ abeja model run-local --image abeja/all-cpu:18.10 --handler main:handler --input cat.jpeg
{
    "result": [
        { "label":"cat", "probability":"0.85" },
        { "label":"dog", "probability":"0.15" }
    ]
}

Notify incorrect JSON input as an exception

By throwing context.exceptions.InvalidInput as an exception, we inform the request executor that it is invalid input. In this case, the example where data does not exist in total_without_tax of input JSON is made invalid input.

def handler(iter, context):
    for line in iter:
        if 'total_without_tax' not in line:
            raise context.exceptions.InvalidInput('total_without_tax not in the request')
        total_without_tax = line['total_without_tax']
        total_without_tax = int(total_without_tax)
        total = total_without_tax * 1.08
        yield {'total': total}

It is the execution result of the run-local command. It can be confirmed that an error is output for an incorrect input.

$ abeja model run-local --image abeja/all-cpu:18.10 --handler tax:handler --input request.json --quiet
[error] failed to send request : 400 Client Error: BAD REQUEST for url: http://localhost:49873/

 ------ Local Server Error ------
{"log_id": "de614348-365b-446f-99c2-44f58d57eefd", "log_level": "INFO", "timestamp": "2018-05-10T01:27:15.416427+00:00", "source": "model:run.run.203", "requester_id": "-", "message": "start executing model. version:0.9.7", "exc_info": null}
{"log_id": "ef77beac-5384-442e-b345-fde112a2952b", "log_level": "INFO", "timestamp": "2018-05-10T01:27:15.416955+00:00", "source": "model:run.run.218", "requester_id": "-", "message": "start installing packages from requirements.txt", "exc_info": null}
{"log_id": "8024e5b7-c4da-4298-8e13-7fbace8c986d", "log_level": "INFO", "timestamp": "2018-05-10T01:27:15.417698+00:00", "source": "model:run.run.224", "requester_id": "-", "message": "requirements.txt not found, skipping", "exc_info": null}
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
{"log_id": "1e730acd-4b6b-4f24-98b5-0c874c122a11", "log_level": "INFO", "timestamp": "2018-05-10T01:27:16.031969+00:00", "source": "model:http_view.endpoint.91", "requester_id": "__requester-id__", "message": "start request", "exc_info": null}
{"log_id": "d6d12c55-57e1-4ca1-bf31-9e6a1b7cb34a", "log_level": "INFO", "timestamp": "2018-05-10T01:27:16.032966+00:00", "source": "model:http_view.endpoint.132", "requester_id": "__requester-id__", "message": "UserException::InvalidInput total_without_tax not in the request", "exc_info": null}
172.17.0.1 - - [10/May/2018 01:27:16] "POST / HTTP/1.1" 400 -

Updated on 01 May 2018