Nuclio Deploying Functions

Published by onesixx on 23-02-1623-02-16

Builtin Model 모델 배포

Serverless tutorial by CVAThttps://opencv.github.io/cvat/docs/manual/advanced/serverless-tutorial/

CVAT에서 이미 만들어진 (serverless) Function을 바로 배포만 하면된다.

cvat 프로젝트를 생성

여기에 serverless function과 model을 저장

$ nuctl create project cvat

Deploy function

CVAT에서 제공하는 배포가능한 serverless function 및 모델 ( https://github.com/openvinotoolkit/cvat/tree/develop/serverless)

$ nuctnuctl deploy \
  --project-name cvat \
  --path     serverless/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio \
  --volume   `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

로컬 환경에서 배포하는 경우에는 —-platform local 옵션을 추가
–http-trigger-service-type NodePort

# 다른 방법
$ serverless/deploy_cpu.sh \
serverless/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio

$ serverless/deploy_gpu.sh \
serverless/tensorflow/matterport/mask_rcnn

23.02.09 17:11:16.786                     nuctl (I) Deploying function {"name": ""}
23.02.09 17:11:16.787                     nuctl (I) Building {"builderKind": "docker", "versionInfo": "Label: 1.8.14, Git commit: cbb0774230996a3eb4621c1a2079e2317578005b, OS: linux, Arch: amd64, Go version: go1.17.8", "name": ""}
23.02.09 17:11:17.014                     nuctl (I) Staging files and preparing base images
23.02.09 17:11:17.015                     nuctl (W) Python 3.6 runtime is deprecated and will soon not be supported. Please migrate your code and use Python 3.7 runtime (`python:3.7`) or higher
23.02.09 17:11:17.015                     nuctl (I) Building processor image {"registryURL": "", "imageName": "cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest"}
23.02.09 17:11:17.015     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.8.14-amd64"}
23.02.09 17:11:20.175     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
23.02.09 17:11:24.389            nuctl.platform (I) Building docker image {"image": "cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest"}
23.02.09 17:17:55.663            nuctl.platform (I) Pushing docker image into registry {"image": "cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest", "registry": ""}
23.02.09 17:17:55.663            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest"}
23.02.09 17:17:55.663                     nuctl (I) Build complete {"result": {"Image":"cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest","UpdatedFunctionConfig":{"metadata":{"name":"openvino-mask-rcnn-inception-resnet-v2-atrous-coco","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"openvino","name":"Mask RCNN","spec":"[\n  { \"id\": 1, \"name\": \"person\" },\n  { \"id\": 2, \"name\": \"bicycle\" },\n  { \"id\": 3, \"name\": \"car\" },\n  { \"id\": 4, \"name\": \"motorcycle\" },\n  { \"id\": 5, \"name\": \"airplane\" },\n  { \"id\": 6, \"name\": \"bus\" },\n  { \"id\": 7, \"name\": \"train\" },\n  { \"id\": 8, \"name\": \"truck\" },\n  { \"id\": 9, \"name\": \"boat\" },\n  { \"id\":10, \"name\": \"traffic_light\" },\n  { \"id\":11, \"name\": \"fire_hydrant\" },\n  { \"id\":13, \"name\": \"stop_sign\" },\n  { \"id\":14, \"name\": \"parking_meter\" },\n  { \"id\":15, \"name\": \"bench\" },\n  { \"id\":16, \"name\": \"bird\" },\n  { \"id\":17, \"name\": \"cat\" },\n  { \"id\":18, \"name\": \"dog\" },\n  { \"id\":19, \"name\": \"horse\" },\n  { \"id\":20, \"name\": \"sheep\" },\n  { \"id\":21, \"name\": \"cow\" },\n  { \"id\":22, \"name\": \"elephant\" },\n  { \"id\":23, \"name\": \"bear\" },\n  { \"id\":24, \"name\": \"zebra\" },\n  { \"id\":25, \"name\": \"giraffe\" },\n  { \"id\":27, \"name\": \"backpack\" },\n  { \"id\":28, \"name\": \"umbrella\" },\n  { \"id\":31, \"name\": \"handbag\" },\n  { \"id\":32, \"name\": \"tie\" },\n  { \"id\":33, \"name\": \"suitcase\" },\n  { \"id\":34, \"name\": \"frisbee\" },\n  { \"id\":35, \"name\": \"skis\" },\n  { \"id\":36, \"name\": \"snowboard\" },\n  { \"id\":37, \"name\": \"sports_ball\" },\n  { \"id\":38, \"name\": \"kite\" },\n  { \"id\":39, \"name\": \"baseball_bat\" },\n  { \"id\":40, \"name\": \"baseball_glove\" },\n  { \"id\":41, \"name\": \"skateboard\" },\n  { \"id\":42, \"name\": \"surfboard\" },\n  { \"id\":43, \"name\": \"tennis_racket\" },\n  { \"id\":44, \"name\": \"bottle\" },\n  { \"id\":46, \"name\": \"wine_glass\" },\n  { \"id\":47, \"name\": \"cup\" },\n  { \"id\":48, \"name\": \"fork\" },\n  { \"id\":49, \"name\": \"knife\" },\n  { \"id\":50, \"name\": \"spoon\" },\n  { \"id\":51, \"name\": \"bowl\" },\n  { \"id\":52, \"name\": \"banana\" },\n  { \"id\":53, \"name\": \"apple\" },\n  { \"id\":54, \"name\": \"sandwich\" },\n  { \"id\":55, \"name\": \"orange\" },\n  { \"id\":56, \"name\": \"broccoli\" },\n  { \"id\":57, \"name\": \"carrot\" },\n  { \"id\":58, \"name\": \"hot_dog\" },\n  { \"id\":59, \"name\": \"pizza\" },\n  { \"id\":60, \"name\": \"donut\" },\n  { \"id\":61, \"name\": \"cake\" },\n  { \"id\":62, \"name\": \"chair\" },\n  { \"id\":63, \"name\": \"couch\" },\n  { \"id\":64, \"name\": \"potted_plant\" },\n  { \"id\":65, \"name\": \"bed\" },\n  { \"id\":67, \"name\": \"dining_table\" },\n  { \"id\":70, \"name\": \"toilet\" },\n  { \"id\":72, \"name\": \"tv\" },\n  { \"id\":73, \"name\": \"laptop\" },\n  { \"id\":74, \"name\": \"mouse\" },\n  { \"id\":75, \"name\": \"remote\" },\n  { \"id\":76, \"name\": \"keyboard\" },\n  { \"id\":77, \"name\": \"cell_phone\" },\n  { \"id\":78, \"name\": \"microwave\" },\n  { \"id\":79, \"name\": \"oven\" },\n  { \"id\":80, \"name\": \"toaster\" },\n  { \"id\":81, \"name\": \"sink\" },\n  { \"id\":83, \"name\": \"refrigerator\" },\n  { \"id\":84, \"name\": \"book\" },\n  { \"id\":85, \"name\": \"clock\" },\n  { \"id\":86, \"name\": \"vase\" },\n  { \"id\":87, \"name\": \"scissors\" },\n  { \"id\":88, \"name\": \"teddy_bear\" },\n  { \"id\":89, \"name\": \"hair_drier\" },\n  { \"id\":90, \"name\": \"toothbrush\" }\n]\n","type":"detector"}},"spec":{"description":"Mask RCNN inception resnet v2 COCO via Intel OpenVINO","handler":"main:handler","runtime":"python:3.6","env":[{"name":"NUCLIO_PYTHON_EXE_PATH","value":"/opt/nuclio/common/openvino/python3"}],"resources":{"requests":{"cpu":"25m","memory":"1Mi"}},"image":"cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"myHttpTrigger","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/raid/templates/cvat/serverless/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat.openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco","baseImage":"openvino/ubuntu18_dev:2020.2","directives":{"postCopy":[{"kind":"RUN","value":"apt update && DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y python3-skimage"},{"kind":"RUN","value":"pip3 install \"numpy<1.16.0\""}],"preCopy":[{"kind":"USER","value":"root"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/bin/pip"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name mask_rcnn_inception_resnet_v2_atrous_coco -o /opt/nuclio/open_model_zoo"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo"}]},"codeEntryType":"image"},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":120,"securityContext":{},"eventTimeout":"60s"}}}}
23.02.09 17:17:55.671                     nuctl (I) Cleaning up before deployment {"functionName": "openvino-mask-rcnn-inception-resnet-v2-atrous-coco"}
23.02.09 17:17:56.545            nuctl.platform (I) Waiting for function to be ready {"timeout": 120}
23.02.09 17:17:58.560                     nuctl (I) Function deploy complete {"functionName": "openvino-mask-rcnn-inception-resnet-v2-atrous-coco", "httpPort": 49156, "internalInvocationURLs": ["172.17.0.7:8080"], "externalInvocationURLs": []}

모델확인
$ nuctl get functions
  NAMESPACE |                        NAME                        | PROJECT | STATE | REPLICAS | NODE PORT    
  nuclio    | openvino-mask-rcnn-inception-resnet-v2-atrous-coco | cvat    | ready | 1/1      |     49156

Example using Python

https://nuclio.io/docs/latest/examples/#python-examples
Nuclio==>https://github.com/nuclio/nuclio/tree/master/hack/examples#python-examples


Hello World	helloworld	A simple function that showcases unstructured logging and a structured response.
Encrypt	encrypt	A function that uses a 3rd-party Python package to encrypt the event body, and showcases build commands for installing both OS-level and Python packages.
FaceRecognizer	face	A function that uses Microsoft’s face API, configured with function environment variables. The function uses 3rd-party Python packages, which are installed by using an inline configuration.
Sentiment Analysis	sentiments	A function that uses the vaderSentiment library to classify text strings into a negative or positive sentiment score.
TensorFlow	tensorflow	A function that uses the inception model of the TensorFlow open-source machine-learning library to classify images. The function demonstrates advanced uses of Nuclio with a custom base image, third-party Python packages, pre-loading data into function memory (the AI Model), structured logging, and exception handling.

Hello World

Function-Configuration

function.yaml, main.py, model_handler.py 준비

https://github.com/nuclio/nuclio/blob/development/docs/reference/function-configuration/function-configuration-reference.md

apiVersion: "nuclio.io/v1beta1"
kind: "NuclioFunction"
metadata:
  name: hiworld
spec:
  # image: xxx:latest
  description: For Test Showcases unstructured logging and a structured response.
  runtime: "python"
  handler: main:handler
  minReplicas: 1
  maxReplicas: 1

#from model_handler import ModelHandler

def init_context(context):
    context.logger.info("Init context...  0%")
    # model = ModelHandler()
    # context.user_data.model = model
    context.logger.info("Init context...100%")

def handler(context, event):
    context.logger.info('This is an unstructured log')

    return context.Response(
        body='hihi, from nuclio :]',
        headers={},
        content_type='text/plain',
        status_code=200
    )

# import cv2
# import numpy as np
import os
#from model_loader import ModelLoader

#class ModelHandler:
    # def __init__(self):
    # def handle(self, image, points):

# class AttributesExtractorHandler:
    # def __init__(self):
    # def infer(self, image):

deploy functions

(mmlab38) ~/my/git/mmdetection/cvat$

# $ nuctl deploy hiworld \
$ nuctl deploy  \
  --path     serverless/sixx/hiworld/nuclio \
  --volume   `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

23.02.16 13:44:32.874                     nuctl (I) Deploying function {"name": "hiworld"}
23.02.16 13:44:32.874                     nuctl (I) Building {"builderKind": "docker", "versionInfo": "Label: 1.8.14, Git commit: cbb0774230996a3eb4621c1a2079e2317578005b, OS: linux, Arch: amd64, Go version: go1.17.8", "name": "hiworld"}
23.02.16 13:44:33.090                     nuctl (I) Staging files and preparing base images
23.02.16 13:44:33.091                     nuctl (I) Building processor image {"registryURL": "", "imageName": "nuclio/processor-hiworld:latest"}
23.02.16 13:44:33.091     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.8.14-amd64"}
23.02.16 13:44:36.511     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
23.02.16 13:44:40.783            nuctl.platform (I) Building docker image {"image": "nuclio/processor-hiworld:latest"}
23.02.16 13:45:33.889            nuctl.platform (I) Pushing docker image into registry {"image": "nuclio/processor-hiworld:latest", "registry": ""}
23.02.16 13:45:33.889            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "nuclio/processor-hiworld:latest"}
23.02.16 13:45:33.889                     nuctl (I) Build complete {"result": {"Image":"nuclio/processor-hiworld:latest","UpdatedFunctionConfig":{"metadata":{"name":"hiworld","namespace":"nuclio","labels":{"nuclio.io/project-name":"default"}},"spec":{"description":"For Test Showcases unstructured logging and a structured response.","handler":"hiworld:handler","runtime":"python:3.7","resources":{"requests":{"cpu":"25m","memory":"1Mi"}},"image":"nuclio/processor-hiworld:latest","minReplicas":1,"maxReplicas":1,"targetCPU":75,"triggers":{"default-http":{"class":"","kind":"http","name":"default-http","maxWorkers":1}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/raid/templates/cvat/serverless/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"codeEntryType":"image"},"platform":{},"readinessTimeoutSeconds":120,"securityContext":{},"eventTimeout":""}}}}
23.02.16 13:45:33.896                     nuctl (I) Cleaning up before deployment {"functionName": "hiworld"}
23.02.16 13:45:34.728            nuctl.platform (I) Waiting for function to be ready {"timeout": 120}
23.02.16 13:45:36.755                     nuctl (I) Function deploy complete {"functionName": "hiworld", "httpPort": 49159, "internalInvocationURLs": ["172.17.0.9:8080"], "externalInvocationURLs": []}

$ nuctl get functions
  NAMESPACE |                        NAME     | PROJECT | STATE | REPLICAS | NODE PORT  
  nuclio    | hiworld                         | default | ready | 1/1      |     49158  
  nuclio    | openvino-dextr                  | cvat    | ready | 1/1      |     49153  
  nuclio    | openvino-mask-rcnn-inception... | cvat    | ready | 1/1      |     49156  
  nuclio    | openvino-omz-public-yolo-v3-tf  | cvat    | ready | 1/1      |     49155  
  nuclio    | pth-foolwood-siammask           | cvat    | ready | 1/1      |     49157  
  nuclio    | tf-matterport-mask-rcnn         | cvat    | error | 1/1      |

$ nuctl invoke hiworld \
 --method POST \
 --body '{"hihihi":"world"}' \
 --content-type "application/json"

23.02.16 11:08:15.114    nuctl.platform.invoker (I) Executing function {"method": "POST", "url": "http://:49158", "bodyLength": 17, "headers": {"Content-Type":["application/json"],"X-Nuclio-Log-Level":["info"],"X-Nuclio-Target":["helloworld"]}}
23.02.16 11:08:15.115    nuctl.platform.invoker (I) Got response {"status": "200 OK"}
23.02.16 11:08:15.115                     nuctl (I) >>> Start of function logs
23.02.16 11:08:15.115                helloworld (I) This is an unstrucured log {"time": 1676513295115.0264}
23.02.16 11:08:15.115                     nuctl (I) <<< End of function logs

> Response headers:
Server = nuclio
Date = Thu, 16 Feb 2023 02:08:14 GMT
Content-Type = application/text
Content-Length = 21

> Response body:
Hello, from Nuclio :]

metadata:
  name: hiworld
  labels:
    nuclio.io/project-name: default
spec:
  description: "For Test Showcases unstructured logging and a structured response."
  handler: "hiworld:handler"
  runtime: "python:3.7"
  resources:
    requests:
      cpu: 25m
      memory: 1Mi
  image: "nuclio/processor-hiworld:latest"
  minReplicas: 1
  maxReplicas: 1
  targetCPU: 75
  triggers:
    default-http:
      class: ""
      kind: http
      name: default-http
      maxWorkers: 1
  volumes:
    - volume:
        name: volume-1
        hostPath:
          path: /home/oschung_skcc/my/git/mmdetection/cvat/serverless/common
      volumeMount:
        name: volume-1
        mountPath: /opt/nuclio/common
  build:
    codeEntryType: image
    timestamp: 1676523194
  platform: {}
  readinessTimeoutSeconds: 120
  securityContext: {}
  eventTimeout: ""
  version: 1

모델배포 : custom model – detectron2

https://opencv.github.io/cvat/docs/manual/advanced/serverless-tutorial/#adding-your-own-dl-models

detectron2설치후 Local에서 DL모델 돌려보기

$ python demo/demo.py \
--config-file configs/COCO-Detection/retinanet_R_101_FPN_3x.yaml \
--input 1920px-Cat_poster_1.jpg \
--output out.jpg \
--opts MODEL.WEIGHTS model_final_971ab9.pkl  MODEL.DEVICE cpu

from detectron2.config import get_cfg
from detectron2.data.detection_utils import read_image

from detectron2.engine.defaults            import DefaultPredictor
from detectron2.data.datasets.builtin_meta import COCO_CATEGORIES

CONFIG_FILE = "configs/COCO-Detection/retinanet_R_101_FPN_3x.yaml"
CONFIG_OPTS = ["MODEL.WEIGHTS", "model_final_971ab9.pkl", "MODEL.DEVICE", "cpu"]
CONFIDENCE_THRESHOLD = 0.5
INPUT_PATH  = "1920px-Cat_poster_1.jpg"

def setup_cfg():
    cfg = get_cfg()
    cfg.merge_from_file(CONFIG_FILE)
    cfg.merge_from_list(CONFIG_OPTS)

    cfg.MODEL.RETINANET.SCORE_THRESH_TEST                      = CONFIDENCE_THRESHOLD
    cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST                      = CONFIDENCE_THRESHOLD
    cfg.MODEL.PANOPTIC_FPN.COMBINE.INSTANCES_CONFIDENCE_THRESH = CONFIDENCE_THRESHOLD
    cfg.freeze()
    return cfg

if __name__ == "__main__":
    cfg = setup_cfg()
	# INPUT	
    img = read_image(INPUT_PATH, format="BGR")
    predictor   = DefaultPredictor(cfg)
    predictions = predictor(img)
    
    instances    = predictions['instances']
    pred_boxes   = instances.pred_boxes
    scores       = instances.scores
    pred_classes = instances.pred_classes
	
    for box, score, label in zip(pred_boxes, scores, pred_classes):
        label = COCO_CATEGORIES[int(label)]["name"]
        print(box.tolist(), float(score), label)
	# Video

function.yaml

1) Annotation 위해 CVAT이 활용할 serverless function 준비

https://opencv.github.io/cvat/docs/manual/advanced/serverless-tutorial/#dl-model-as-a-serverless-function

함수의 이름 정하기

파라미터 전체설명 : https://nuclio.io/docs/latest/reference/function-configuration/function-configuration-reference/

metadata.name : pth.facebookresearch.detectron2.retinanet_r101

metadata.annotations.name : 보여지는 이름

meradata.annotation.type: serverless function의 종류 . detector

metadata.annotation.framework: 기냥 정보성.. 특별히 정해진건 없으나..OpenVINO, PyTorch, TensorFlow, etc.

function.yaml

metadata:
  name: pth.facebookresearch.detectron2.retinanet_r101
  namespace: cvat
  annotations:
    name: RetinaNet R101
    type: detector
    framework: pytorch
    spec: |
      [
        { "id": 1, "name": "person" },
        { "id": 2, "name": "bicycle" },

        ...

        { "id":89, "name": "hair_drier" },
        { "id":90, "name": "toothbrush" }
      ]      

spec:
  description: RetinaNet R101 from Detectron2
  runtime: 'python:3.8'
  handler: main:handler
  eventTimeout: 30s

  build:
    image: cvat/pth.facebookresearch.detectron2.retinanet_r101
    baseImage: ubuntu:20.04

    directives:
      preCopy:
        - kind: ENV
          value: DEBIAN_FRONTEND=noninteractive
        - kind: RUN
          value: apt-get update && apt-get -y install curl git python3 python3-pip
        - kind: WORKDIR
          value: /opt/nuclio
        - kind: RUN
          value: pip3 install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
        - kind: RUN
          value: pip3 install 'git+https://github.com/facebookresearch/detectron2@v0.4'
        - kind: RUN
          value: curl -O https://dl.fbaipublicfiles.com/detectron2/COCO-Detection/retinanet_R_101_FPN_3x/190397697/model_final_971ab9.pkl
        - kind: RUN
          value: ln -s /usr/bin/pip3 /usr/local/bin/pip

  triggers:
    myHttpTrigger:
      maxWorkers: 2
      kind: 'http'
      workerAvailabilityTimeoutMilliseconds: 10000
      attributes:
        maxRequestBodySize: 33554432 # 32MB

  platform:
    attributes:
      restartPolicy:
        name: always
        maximumRetryCount: 3
      mountMode: volume

외부에서 해당 모델을 통해 접근을 위한 설정

nuclio function의 포트번호와 cvat network를 지정

spec.triggers.myHttpTrigger.attributes에 특정 port를 추가하고,
spec.platform.attributes 에 도커 컴포즈를 실행했을 때 생성된 도커 네트워크(cvat_cvat)를 추가.

  triggers:
    myHttpTrigger:
      maxWorkers: 2
      kind: 'http'
      workerAvailabilityTimeoutMilliseconds: 10000
      attributes:
        maxRequestBodySize: 33554432
        port: 33000 # 추가포트 

  platform:
    attributes:
      restartPolicy:
        name: always
        maximumRetryCount: 3
      mountMode: volume
      network: cvat_cvat # 추가 도커네트웍

2. 로컬에서 DL모델을 실행하기 위한 소스코드를 Nuclio 플랫폼에 적용

2-1 모델을 메모리에 로딩 (init_context(context)함수를 사용하여)

https://nuclio.io/docs/latest/concepts/best-practices-and-common-pitfalls/#use-init_context-instead-of-global-variable-declarations-or-function-calls

def handler(context, event):
    # context.user_data.my_db_connection.query(...)


def init_context(context):
    # Create the DB connection under "context.user_data"
    # setattr(context.user_data, 'my_db_connection', my_db.create_connection())

    cfg = get_config('COCO-Detection/retinanet_R_101_FPN_3x.yaml')
    cfg.merge_from_list(CONFIG_OPTS)
    
    cfg.MODEL.RETINANET.SCORE_THRESH_TEST = CONFIDENCE_THRESHOLD
    cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = CONFIDENCE_THRESHOLD
    cfg.MODEL.PANOPTIC_FPN.COMBINE.INSTANCES_CONFIDENCE_THRESH = CONFIDENCE_THRESHOLD
    
    cfg.freeze()
    predictor = DefaultPredictor(cfg)
    
    context.user_data.model_handler = predictor

2-2 아래 프로세스를 위해 handler에 entry point를 정의하고, main.py에 넣는다.

accept incoming HTTP requests
run inference
reply with detection results

main.py

import json
import base64
import io
from PIL import Image
import yaml

from detectron2.config import get_cfg
from detectron2.data.detection_utils import convert_PIL_to_numpy
from detectron2.engine.defaults import DefaultPredictor
from detectron2.data.datasets.builtin_meta import COCO_CATEGORIES

CONFIG_FILE = "detectron2/configs/COCO-Detection/retinanet_R_101_FPN_3x.yaml"
CONFIG_OPTS = ["MODEL.WEIGHTS", "model_final_971ab9.pkl", "MODEL.DEVICE", "cpu"]
CONFIDENCE_THRESHOLD = 0.5

def handler(context, event):
    context.logger.info("Run retinanet-R101 model")
    
    data      = event.body
    buf       = io.BytesIO(base64.b64decode(data["image"]))
    threshold = float(data.get("threshold", CONFIDENCE_THRESHOLD)) #0.5))
    image     = convert_PIL_to_numpy(Image.open(buf), format="BGR")

    predictions  = context.user_data.model_handler(image)
    
    instances    = predictions['instances']
    pred_boxes   = instances.pred_boxes
    scores       = instances.scores
    pred_classes = instances.pred_classes
    
    results = []
    for box, score, label in zip(pred_boxes, scores, pred_classes):
        label = COCO_CATEGORIES[int(label)]["name"]
        if score >= threshold:
            results.append({
                "confidence": str(float(score)),
                "label": label,
                "points": box.tolist(),
                "type": "rectangle",
            })

    return context.Response(
        body=json.dumps(results), 
        headers={},
        content_type='application/json', 
        status_code=200)


def init_context(context):
    context.logger.info("Init context...  0%")

    cfg = get_config(CONFIG_FILE) # 'COCO-Detection/retinanet_R_101_FPN_3x.yaml')
    cfg.merge_from_list(CONFIG_OPTS)
    cfg.MODEL.RETINANET.SCORE_THRESH_TEST = CONFIDENCE_THRESHOLD
    cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = CONFIDENCE_THRESHOLD
    cfg.MODEL.PANOPTIC_FPN.COMBINE.INSTANCES_CONFIDENCE_THRESH = CONFIDENCE_THRESHOLD
    cfg.freeze()
    
    predictor = DefaultPredictor(cfg)

    #context.user_data.model_handler = predictor
    setattr(context.user_data, 'model_handler', predictor)
    functionconfig = yaml.safe_load(open("/opt/nuclio/function.yaml"))
    labels_spec    = functionconfig['metadata']['annotations']['spec']
    labels         = {item['id']: item['name'] for item in json.loads(labels_spec)}
    setattr(context.user_data, "labels", labels)    
    
    context.logger.info("Init context...100%")

3. deploy

새로운 serverless 함수를 사용하기 위해서는
(위 Builtin model에서 했던것처럼) nuctl 명령어로 deploy를 해야한다.

function.yaml
main.py
model_handler.py

방법1)

$ nuctl deploy --project-name cvat \
  --path     serverless/pytorch/facebookresearch/detectron2/retinanet/ \
  --volume   `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

방법2)

$ serverless/deploy_cpu.sh \
serverless/pytorch/facebookresearch/detectron2/retinanet/

Issue

https://github.com/opencv/cvat/issues/3457 : Steps for custom model deployment

http://How to upload DL which built by myself? : How to upload DL which built by myself?

https://github.com/opencv/cvat/issues/5551 : Mmdetection MaskRCNN serverless support for semi-automatic annotation

https://github.com/opencv/cvat/issues/4909: Load my own Yolov5 model on cvat by nuclio

mmdetection 모델변환

https://mmdetection.readthedocs.io/en/latest/useful_tools.html#model-conversion

Exporting MMDetection models to ONNX format
https://medium.com/axinc-ai/exporting-mmdetection-models-to-onnx-format-3ec839c38ff

openvino : https://da2so.tistory.com/63

Serving pre-trained ML/DL models
https://docs.mlrun.org/en/stable/tutorial/03-model-serving.html#serving-pre-trained-ml-dl-models

Nuclio Deploying Functions

Builtin Model 모델 배포

Serverless tutorial by CVAThttps://opencv.github.io/cvat/docs/manual/advanced/serverless-tutorial/

cvat 프로젝트를 생성

Deploy function

Example using Python

https://nuclio.io/docs/latest/examples/#python-examples
Nuclio==>https://github.com/nuclio/nuclio/tree/master/hack/examples#python-examples

Hello World

Function-Configuration

deploy functions

모델배포 : custom model – detectron2

https://opencv.github.io/cvat/docs/manual/advanced/serverless-tutorial/#adding-your-own-dl-models

function.yaml

mmdetection 모델변환

onesixx

vision

Nuclio 모델배포 : mmdetect

vision

Nuclio 모델배포 : custom model – Mask

vision

Nuclio 모델배포 : custom model

Nuclio Deploying Functions

Builtin Model 모델 배포

Serverless tutorial by CVAThttps://opencv.github.io/cvat/docs/manual/advanced/serverless-tutorial/

cvat 프로젝트를 생성

Deploy function

Example using Python

https://nuclio.io/docs/latest/examples/#python-examplesNuclio==>https://github.com/nuclio/nuclio/tree/master/hack/examples#python-examples

Hello World

Function-Configuration

deploy functions

모델배포 : custom model – detectron2

https://opencv.github.io/cvat/docs/manual/advanced/serverless-tutorial/#adding-your-own-dl-models

function.yaml

mmdetection 모델변환

onesixx

Related Posts

vision

Nuclio 모델배포 : mmdetect

vision

Nuclio 모델배포 : custom model – Mask

vision

Nuclio 모델배포 : custom model

https://nuclio.io/docs/latest/examples/#python-examples
Nuclio==>https://github.com/nuclio/nuclio/tree/master/hack/examples#python-examples