Sign Language Motion Capture Using Optical Marker-Based Systems

Sign Language (SL) data (usually motion) capture hardware high-level overview:

1. Optical cameras Collect light
    a) RGB / RGB-D Only collects light. Sensitive to variations
        i) Marker-based Calculate position and orientation using markers. Requires marker identification
        ii) Marker-less Have to extract silhouettes/edges to calculate position and orientation
    b) Spectral Emits and collects light. Not too sensitive to variations (e.g. infra-red)
        i) Marker-based Calculate position and orientation using light coming from markers
            1) Active markers Markers are identified distincly by capturing the frequency of the light that they emit
            2) Passive markers Capture reflection from markers. Need to be assigned labels manually
                a) Concave Reflect light in any direction
                b) Flat Poor reflection during affine transformation
        i) Marker-less Have to extract silhouettes/edges to calculate position and orientation
2. Gyroscopes Measure rotation
3. Accelerometers Measure acceleration (e.g. IMU sensors)
4. Flex sensors Measures degree of deformation / bending (e.g. data-glove)

RGB Marker-less →
Region of interest detection, feature extraction, feature tracking
Spectral Passive Marker-Based →
Raw, labeled data
Examples

The choice of hardware is a balance between the precision of the data and degree of restriction of free movement. The best option is to choose based on the context/goal. Acquisition of various expensive heterogeneous hardware is inevitable if one wants to gather accurate data and minimally restrict the signer.

Sign Language (SL) data (usually motion) capture steps:
  1. Prepare environment
    • Set up the hardware (location, orientation, settings). Think whether you need large capture volume or small capture volume. This will affect captured data
    • Determine location of markers (if any). Try to simplify joints (e.g. do not place markers on every joint)
      • Place markers on signer (avoid markers' movements between recordings / signers)
    • Cover all reflecting surfaces
    • Adjust light / keep light constant
  2. If no model, then create Model (best to have signer-specific model)
    • Record Data
    • Post-Processing
      • Label captured markers
      • Handle overlapping markers (avoid reversing markers' identifiers)
      • Fill gaps (linear, cubic, or using neighbouring markers)
      • Remove noise (e.g. reflections)
      • Generate model
  3. If model exists, use the model
    • Record Data
    • Apply Model
    • Post-Processing
      • Label not labeled markers
      • Handle overlapping markers (avoid reversing markers' identifiers)
      • Fill gaps (linear, cubic, or using neighbouring markers)
      • Remove noise (e.g. reflections)