Frame Accurate Video Scrubbing in the Client

There are a variety of desktop applications that allow users to seamlessly scrub back and forth through a video to determine the exact frame in which some action should take place or where the placement of a tag should go. Some popular applications include Final Cut Pro and Adobe Premiere Pro. This sort of smooth functionality is difficult to attain on a website due to the limited resources of a browser client in comparison with a desktop environment. 

In this post, I will walk through how to create a simple frame-accurate video scrubber for video segments in a browser client using the HTML5 video element.

The inspiration for this scrubber implementation comes from this Github post: Frame Accurate Scrubbing, which describes a method that would extract frames on the backend and serve them up to a client for accurate scrubbing.

There are many great open-source packages that make for a simple frame grabbing implementation on the backend, and most of these packages rely on FFMPEG to decode video. 

Since GPL distributes FFmpeg, this is an approach that can be incompatible with projects that aren’t destined to be open source or also distributed under GPL. By using the HTML5 video element to extract frames, we can avoid FFPMEG’s viral GPL license as well as the hassle of writing the video decoder ourselves.

Environment

The examples below show the JSX markup of the development we did in React JS.

Prerequisites

This implementation requires a known framerate for the video you want to use. Additionally, I did this development using a Chrome browser, which supports the MP4 video format that I was using. Make sure that your browser supports your desired video format for HTML5 video.

With that out of the way, let’s get started!

HTML5 Video Element as Video Decoder

Create an HTML5 Video component on your page, but keep it hidden. Store a reference to this video player. Also, create a div to hold the extracted frames.

render() {
  return (
    <div className="video-scrubber">
      <video
        src={"video.mp4"}
        muted={true}
        hidden={true}
        controls={false}        
        onSeeked={this.extractFrame} 
        ref={(element) => { this.videoElement = element }}>
      </video>

      <div className="video-scrubber-frame-container"  
        ref={(element) => this.frameContainerElement = element}>
      </div>  
    </div>
  )    
}

Create canvas representations of the video frames. Seek to one frame at a time using the known framerate of the video. The video component’s onSeeked event triggers the extractFrame function that draws the current video frame onto a canvas element.

loadVideoFrames = () => {
  // Exit loop if desired number of frames have been extracted 
  if (this.frames.length >= frameCount){
    this.setState({
      visibleFrame = 0 
    })

    // Append all canvases to container div
    this.frames.forEach((frame) => {this.frameContainerElement.appendChild(frame)})
    return
  }

  // If extraction hasn’t started, set desired time for first frame 
  if (this.frames.length === 0){
    this.requestedTime = 0
  }
  else{
    this.requestedTime = this.requestedTime + this.frameTimestep
  }

  // Send seek request to video player for the next frame.
  this.videoElement.currentTime = this.requestedTime
}
 
 
extractFrame = () => {
  // Create DOM canvas object
  var canvas = document.createElement('canvas')
  canvas.className = "video-scrubber-frame"
  canvas.height = videoHeight
  canvas.width = videoWidth
      
  // Copy current frame to canvas
  var context = canvas.getContext('2d');
  context.drawImage(this.videoElement, 0, 0, videoWidth, videoHeight);  
  this.frames.push(canvas)

  //  Load the next frame
  loadVideoFrames()
}

Define styling that will layer the canvases on top of each other.

.video-scrubber-frame-container {
    position: relative;
}

.video-scrubber-frame {
    left: 0;
    position: absolute;
    top: 0;
}

Implement Scrubbing Functionality

Add mouse event listeners to handle scrubbing action. Here we create another canvas that will sit on top of all our frames to attach our listeners to.

// Create mouseover canvas
var mouseEventCanvas = document.createElement('canvas')

// Set canvas attributes
mouseEventCanvas.className = "video-scrubber-frame"
mouseEventCanvas.height = videoHeight
mouseEventCanvas.width = videoWidth
mouseEventCanvas.onmousemove = this.onMouseDown
mouseEventCanvas.onmousedown = this.onMouseDown

// We set the z-index to be greater than that of the other frames
mouseEventCanvas.style['z-index'] = 2

// Append to document
this.frameContainerElement.appendChild(mouseEventCanvas)

Toggle z-index of visible canvases according to mouse positioning over the frames. For our scrubber, we are listening for a mousedown event that moves on the frames. As a user drags their mouse across the frames, we toggle the visible frame value stored in our React component state.

onMouseDown = (event) => {
  // Bail early if we aren't clicking and dragging
  if (event.type === "mousemove" && event.buttons === 0) {
    return 
  }

  event.preventDefault()
  
  const frameRect = this.mouseEventCanvas.getBoundingClientRect()
  const x = event.clientX - frameRect.left
  const y = event.clientY - frameRect.top

  if (event.buttons === 1){ // Left button down
    var frameRequested = Math.floor((x / frameRect.width) 
* this.fullResolutionFrames.length)      
    if (frameRequested !== this.state.visibleFrame && frameRequested >= 0){
      this.setState({
        visibleFrame: frameRequested,
      })
    }
  }
}

We check for a changed visibleFrame value in the React component lifecycle and change the z-indexes of the appropriate frames.

componentWillUpdate(nextProps, nextState){
  // set the visible frame
  if (nextState.visibleFrame !== this.state.visibleFrame){
    if (this.state.visibleFrame < this.frames.length){
      this.frames[this.state.visibleFrame].style['z-index'] = 0 
    }   
    if (nextState.visibleFrame < this.frames.length){   
      this.frames[nextState.visibleFrame].style['z-index'] = 1 
    }
  }
}

There you have it! A simple video scrubber in a client browser.

Learn more about DMC's Web Application Development services.