Model handler function FOR 19.04 +

This page explaines the handler functions that can be used in images after 19.04. See here for the handler functions available in the 18.10 image.

Model handler is a function in the code that are called when the model is executed. The handler function uses the following syntax structure.

def handler(request, context):
    ...
    return response

• request : Input data is stored in this parameter. This parameter is dict. It behaves like file-like object.
• context : Runtime metadata etc. are stored in this parameter.

Handler arguments

Input Data

Input data is passed to the first argument of the handler function. The format of the data included in the input data differs for each Content-Type.

Other than multipart/form-data

The following items can be acquired from the input data.

Key	Description
http_method	Request HTTP Method
content_type	Content-Type
headers	Request Header. It is a list of dicts containing header names and values, all header names are converted to lower case. Some headers will be removed, including Authorization header
http_headers	Header information which you can access by dict format.Included header information is same as headers
read()	Request body

You can get each value as follows.

>>> request['http_method']
'POST'
>>> request['content_type']
'application/json'
>>> pprint.pprint(request['http_headers']['accept-encoding'])
'gzip, deflate, br'
>>> request.read()
b'{"foo": "bar"}'

The request body is always read as binary data. GET is Request parameter,POST PUT are request body.

multipart/form-data

The following items can be acquired from the input data.

Key	Description
http_method	Request HTTP Method
content_type	Content-Type
headers	Request Header. It is a list of dicts containing header names and values, all header names are converted to lower case. Some headers will be removed, including Authorization header
http_headers	Header information which you can access by dict format.Included header information is same as headers
contents	Contents

Contents is an array ofContent for each part.
The following elements can be obtained from each Content.

Key	Description
content_type	Content-Type
form_name	Form Name
file_name	File Name
read()	Contents body

You can get each value as follows.

>>> request['http_method']
'POST'
>>> request['content_type']
'multipart/form-data'
>>> contents = request['contents']
>>> contents[0]['content_type']
'application/json'
>>> contents[0]['form_name']
'file1'
>>> contents[0].read()
b'{"foo": "bar"}'
>>> contents[1]['content_type']
'image/jpeg'
>>> contents[1]['file_name']
'cat.jpg'
...

Context data

The metadata of the runtime will be passed to the second argument of the handler function. This variable is an empty dict.

Handler environment variables

The handler environment can use the following environment variables.

Environment variable name	Description
ABEJA_ORGANIZATION_ID	ID of the organization
ABEJA_MODEL_ID	ID of the model
ABEJA_MODEL_VERSION_ID	ID of the model version
ABEJA_DEPLOYMENT_ID	ID of the deployment
ABEJA_SERVICE_ID	Service ID
HANDLER	Handler path
TRAINING_JOB_DEFINITION_NAME	Training job definition name
TRAINING_JOB_ID	ID of the Training job
ABEJA_TRAINING_RESULT_DIR	Path to the directory where learning job results are stored

In addition to the above, environment variables that can be specified when creating code version/service/trigger/run can also be used.
For more information on user-specifiable environment variables, see here.

Using learning results in a model handler

ABEJA Platform can create model versions based on learning results.

When using the output of learning in the model handler function, it must be stored in the directory specified by the environment variable ABEJA_TRAINING_RESULT_DIR during learning. The results output during learning are expanded to the directory stored in the environment variable ABEJA_TRAINING_RESULT_DIR when the model handler function is executed.

Handler return values

To output data, return a dict containing the following elements.

Key	Description
status_code	HTTP Status Code
content_type	Content-Type
content	Response body
metadata	Dict of the element you want to return in the response header

If you do not specify status_code,http.HTTPStatus.OK is used as the default value. If content_type is not specified,text / plain is used as the default value.

For example, the value is returned as follows.

import http

def handler(request, context):

    # some inference process...

    content = {
        'transaction_id': 1234567890,
        'category_id': 10,
        'predictions': predictions,
    }
    return {
        'status_code': http.HTTPStatus.OK,
        'content_type': 'application/json; charset=utf8',
        'content': content
    }

Implementation example of model handler function

The following is a simple implementation example of a model handler function.

Input JSON data and return JSON data

This is an example of performing a simple calculation on the input JSON data and returning a JSON data response.

Get the input data from the input data as follows.

import json
import http

def handler(request, context):
    req_data = json.load(request)
    total_without_tax = req_data['total_without_tax']
    total_without_tax = int(total_without_tax)
    total = total_without_tax * 1.08

    return {
        'status_code': http.HTTPStatus.OK,
        'content_type': 'application/json; charset=utf8',
        'content': {'total': total}
    }

Execution result of run-local command. The output result with request.json as input is displayed.

$ abeja model run-local --image abeja/all-cpu:19.04 --handler main:handler --input request.json --quiet
{
    "total": 108.0
}

Return an image as an input

This is an example of returning the input image flipped horizontally.

As with JSON, the input data is obtained from the input data as follows.

The inverted input image is returned as a bytes type in thecontent. In this example, since the output is a JPEG image, set content_type toimage/jpeg as follows.

import cv2
import http
import numpy as np
from io import BytesIO
from PIL import Image

def handler(request, context):
    data = request.read()
    imgfile = BytesIO(data)
    img_pil = Image.open(imgfile)
    img_numpy = np.asarray(img_pil)
    flipped_array = cv2.flip(img_numpy, 1)
    flipped_image = Image.fromarray(flipped_array)
    retBytes = BytesIO()
    flipped_image.save(retBytes, format='JPEG', quality=95)
    retBytes.seek(0)

    return {
        'status_code': http.HTTPStatus.OK,
        'content_type': 'image/jpeg',
        'content': retBytes
    }

Also, since Pillow is used as the image processing library,Pillow installation is specified in requirements.txt (orPipfile).

For requirements.txt

Pillow==6.0.0

For Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
Pillow = "==6.0.0"

[requires]
python_version = "3.6"

Execution result of run-local command. An image obtained by inverting the input cat.jpg is output as flipped_cat.jpg.
Please note that if you do not specify the --quiet option, the log will also be output and a JPEG file will not be generated.

$ abeja model run-local -h main:handler -i abeja/all-cpu:19.04 --input cat.jpg --quiet > flipped_cat.jpg

Return JSON from images using learning results

The inference result of the model using the HDF5 file output as the learning result is returned in JSON.

Read and infer HDF5 file from the learning result expanded to ABEJA_TRAINING_RESULT_DIR. In this case, it is necessary to specify the same file name as the HDF5 file saved in the output directory during learning.

import os
import numpy as np
from io import BytesIO
from keras.models import load_model
from PIL import Image

model = load_model(os.path.join(os.environ.get('ABEJA_TRAINING_RESULT_DIR', '.'), 'model.h5'))

def handler(request, ctx):
    data = request.read()
    imgfile = BytesIO(data)
    img_pil = Image.open(imgfile)
    img_numpy = np.asarray(img_pil)

    ...

    result = model.predict(img)

    # CONVERT RESULT FOR RESPONSE
    ...

    return {
        'status_code': http.HTTPStatus.OK,
        'content_type': 'application/json; charset=utf8',
        'content': result
    }

Execution result of run-local command. An inference result with the input cat.jpg as an input is returned.

The environment variable ABEJA_TRAINING_RESULT_DIR is set toabejainc_training_result as an initial value. Note that if not specified, HDF5 files will be searched under <current directory>/abejainc_training_result/

$ abeja model run-local -h main:handler -i abeja/all-cpu:19.04 --input cat.jpg --environment ABEJA_TRAINING_RESULT_DIR:. --quiet
{
    "result": [
        { "label":"cat", "probability":"0.85" },
        { "label":"dog", "probability":"0.15" }
    ]
}

Notify invalid JSON input

Set status_code to notify requester that the input is invalid. In this case, invalid input is when there is no data in the total_without_tax of the input JSON data.

import json
import http

def handler(request, context):
    req_data = json.load(request)
    if 'total_without_tax' not in req_data:
        return {
            'status_code': http.HTTPStaus.BAD_REQUEST,
            'content_type': 'application/json; charset=utf8',
            'content': {'message': 'total_without_tax is need'}
        }

    ...

Execution result of run-local command. An inference result with the input request.json as input is returned. It can be confirmed that an error is output for invalid input.

$ abeja model run-local --image abeja/all-cpu:19.04 --handler main:handler --input request.json --quiet
[error] failed to send request : 400 Client Error: Bad Request for url: http://localhost:62910/

 ------ Local Server Error ------
{"level":"info","msg":"proxy version: [0.4.1] start run.","time":"2019-05-23T09:29:06Z"}
{"level":"debug","msg":"call handleSignal","time":"2019-05-23T09:29:06Z"}
{"level":"debug","msg":"handleSignal: create channel for signal","time":"2019-05-23T09:29:06Z"}
{"level":"debug","msg":"runtime bootstrapping yet...","time":"2019-05-23T09:29:06Z"}
{"address":":5000","level":"debug","msg":"start listen.","time":"2019-05-23T09:29:06Z"}
{"level":"debug","msg":"handleSignal: channel for signal created","time":"2019-05-23T09:29:06Z"}
{"level":"debug","msg":"waiting signal...","time":"2019-05-23T09:29:06Z"}
{"log_id": "b94bba13-7046-4697-8417-8ab2f5020387", "log_level": "INFO", "timestamp": "2019-05-23T09:29:07.194826+00:00", "source": "runtime:run.main.26", "requester_id": "-", "message": "start executing model with abeja-runtime-python36 (version: 0.1.4)", "exc_info": null}
INFO: requirements.txt/Pipfile not found, skip installing dependencies process
INFO: Loading user model handler with main:handler...
{"level":"debug","msg":"runtime started","time":"2019-05-23T09:29:07Z"}
{"level":"debug","msg":"response body length = 157","time":"2019-05-23T09:29:08Z"}
{"level":"debug","msg":"response body = {\"content_type\": \"application/json; charset=utf8\", \"metadata\": {\"x-abeja-sys-meta-runtime-version\": \"0.1.4\"}, \"path\": \"/tmp/tmpgl8iizco\", \"status_code\": 400}","time":"2019-05-23T09:29:08Z"}