CoreML WWDC2020 New Features

New features at WWDC2020

Slide version: https://www.slideshare.net/AbbyLai1/coreml-new-features-in-2020

Vision

New features in Vision.framework

Features

  • Hand Pose Detection
  • Body Pose Detection
  • Trajectory Detection
  • Contour Detection
  • Optical Flow

Limitation

  • iOS 14.0+
  • masOS 11.0+

Hand Pose Detection

[Apple documentation] VNDetectHumanHandPoseRequest

[WWDC] Detect Body and Hand Pose with Vision

  • 4 landmarks for each finger
  • 1 landmarks for the wrist
  • 21 landmarks total
21 landmarks
WWDC2020 demo

Body Pose Detection

[Apple documentation] Detecting Human Body Poses in Images

[WWDC] Detect Body and Hand Pose with Vision

  • 19 body points

WWDC2020 Sample Code

Building a Feature-Rich App for Sports Analysis

Trajectory Detection

An observation that describes a detected trajectory

[Apple documentation] Identifying Trajectories in Video

  • Detect multiple, simultaneous trajectories of objects in a video sequence
  • Requires a stable scene
    meaning the camera is mounted on a tripod, and the camera and background remain stationary.

Response Parameters

  • The coefficients of the parabolic equation
  • The centroid points of the detected contour along the trajectory
  • The centroids of the calculated trajectory from the detected points

Contour Detection

Detect the contours of the edges of an image

[Apple documentation] VNDetectContoursRequest

Response Parameters

  • normalizedPath — It returns the path of detected contours in normalized coordinates.
  • contourCount — The number of detected contours returned by the Vision request.
  • topLevelContours — An array of VNContours that aren’t enclosed inside any contour.
  • contour(at:) — Using this function, we can access a child contour by passing its index or IndexPath.
  • confidence — The level of confidence in the overall VNContoursObservation.

Optical Flow

[Apple documentation] VNGenerateOpticalFlowRequest

  • An object that determines directional change vectors for each pixel in the targeted image to transform the image into the one processed by the request handler.
  • Both images must have the same dimensions
  • Setting a region of interest limits the region in which the analysis occurs; however, the system reports the resulting observation at full resolution.
  • Optical flow requests are resource intensive.

Create ML

Features

  • Action Classification
  • Style Transfer

Action Classification

[Apple documentation] Build an Action Classifier with Create ML (Start from 19:30)

Style Transfer

A model you train to apply an image’s style to other images or video

[Apple documentation] Build Image and Video Style Transfer models in Create ML

Style transfer wit ARKit (Video start from 08:30)

iOS Developer