debug-local

Description

The environment specified by --image is built locally, and the learning code that exists locally is executed on that environment.

Notes

In order to use the debug-local command, docker must be installed.

Synopsis

$ abeja training debug-local [--help]
Usage: abeja training debug-local [OPTIONS]

  Local train commands

Options:
  -h, --handler TEXT              Training handler  [required]
  -i, --image TEXT                Specify base image name and tag in the
                                  "name:tag" format  [required]
  -o, --organization_id, --organization-id TEXT
                                  Organization ID, organization_id of current
                                  credential organization is used by default
                                  [required]
  --datasets DATASETPARAMSTRING   Datasets name
  -e, --environment ENVIRONMENTSTRING
                                  Environment variables
  -v, --volume VOLUMEPARAMSTRING  Volume driver options, ex) /path/source/on/h
                                  ost:/path/destination/on/container
  --no-cache, --no_cache          Not use built cache
  --v1                            Specify if you use old custom runtime image
  --runtime TEXT                  Runtime, equivalent to docker run
                                  `--runtime` option
  --build-only                    Build a docker image only. Not run train
                                  command.
  -q, --quiet                     Suppress info logs
  --config PATH                   Read Configuration from PATH. By default
                                  read from `training.yaml`
  --help                          Show this message and exit.

Options

-h, --handler

Specify the path of the function to be called. If --handler main: handler is specified,handler defined in the main.py file is called. If the file to call is placed directly under the src directory, it will be src.main: handler.

-i, --image

Specify the image to use. See here for images that can be specified.

-o, --organization_id, --organization-id

Specify the organization ID. The organization ID is registered in the environment variable with the key ABEJA_ORGANIZATION_ID. It can be referenced from the learning code to be executed.

--datasets

Specify the data set to be used in the following format. {dataset_name}:{dataset_id} The registered dataset can be referenced from the context given as an argument of the training code.

( version 0.14.0 or later ) With training.yaml, values defined as datasets in training.yaml are set by default.
This option can overwrite datasets in training.yaml.

-e, --environment

Specify an environment variable. Registered environment variables can be referenced from the code. e.g.）IMAGE_WIDTH:100
For more information on user-specifiable environment variables, see here.

( version 0.14.0 or later ) With training.yaml, values defined as environment ( params ) in training.yaml are set by default.
This option can overwrite environment ( params ) in training.yaml.

-v, --volume

This corresponds to the --volume option of the docker run command. In the format --volume/path/source/on/host:/path/destination/on/container, specify the host-side path to be mounted and the container-side path to be mounted. It is possible to specify more than one. The --read-only option of the docker run command is not supported.

--runtime

This corresponds to the --runtime option of the docker run command. The debug-local command runs the learning job as a container. The --runtime option specifies the runtime for starting the container.

For example, if nvidia-docker2 is installed, you can run learning using GPU by specifying --runtime nvidia.

--build-only

Only build the image, no learning itself. The built image will be named [specified image name]/[specified image tag]/train-local-model.

--v1

This option should be given when using 18.10 custom images.

--no_cache, --no-cache

Rebuild and recreate the image.

-q, --quiet

Suppress info log output.

-d option has been disabled since version 1.1.0.

Example

Debug learning locally

Premise：

• Assume the following status.

$ cat main.py
def handler(context):
    dataset_alias = context.datasets
    dataset_id = dataset_alias['dataset_name']
    ...

Command:

$ abeja training debug-local \
    -h main:handler \
    -i abeja/all-cpu:18.10 \
    --organization_id 1234567890123 \
    --datasets dataset_1:1000000000

Output:

[info] preparing image : abeja/all-cpu:18.10
[info] building image
INFO: start installing packages from requirements.txt
INFO: packages are installed from requirements.txt
...

Handler environment variables

The handler function executed by debug-local command can use the following environment variables.

Environment variable name	Description
ABEJA_ORGANIZATION_ID	ID of the organization By default, organization ID of the one in ~/.abeja/config is set ( version 0.12.4 or later ). This value can be overwritten by setting --organization_id option.
HANDLER	Handler path
DATASETS	Dataset information
ABEJA_PLATFORM_USER_ID	User ID By default, value in ~/.abeja/config as abeja-platform-user is set ( version 0.12.4 or later ). This value can be overwritten by setting ABEJA_PLATFORM_USER_ID with --environment option.
ABEJA_PLATFORM_PERSONAL_ACCESS_TOKEN	Personal Access Token By default, value in ~/.abeja/config as personal-access-token is set ( version 0.12.4 or later ). This value can be overwritten by setting ABEJA_PLATFORM_PERSONAL_ACCESS_TOKEN with --environment option.