Linguistic Visual Feature Vector

Signs in sign languages can be described using the parameters, such as 1) DEZ - shape of the hand, 2) ORI - orientation of the hand, 3) SIG - movement of the hand, 4) TAB - location of the hand, and 5) NMS - non-manual signs, such as facial expressions, body posture, and similar. Some notation systems, such as HamNoSys have been developed to capture these parameters in their phonetic transcription systems.
DEZ
ORI
  • Extended finger direction
  • Extended finger direction in relation to the body (away/towards the body/etc)
  • Palm orientation
  • Palm orientation in relation to the body (palm up/down/towards body/etc)
SIG
  • Straight (can be targeted)
  • Curved with the direction of the curve (can be targeted)
  • Zigzag, wavy
  • Circular
  • In-place movement (replacement of the handshape, orientation)
TAB
  • Proximity of the hand to a body part
  • Side of the body part
NMS
  • Will be handled in the future


Example

  • Speeds graph for a closed interval from frame N to frame M
  • Red point signifies both hands' movement with major change in the speed
  • Green point signifies when the change in the movement started taking place (backtracking)
ORI
  • Yellow line shows the palm orientation
  • Black line shows the direction of the extended finger
SIG
  • Yellow lines show the hands' movement trajectory on the closed interval from frame N to frame M
TAB
  • Dots on the black background show skeleton keypoints, identified with the OpenPose library
  • Orange dots are the tracked centroids of the palms
  • White dots are the triggered body parts in proximity of the centroids of the palms (according to the proximity matrix heatmap)


Live Demo


https://github.com/mocialov/LinguisticVisualFeatureVector