It is very often that I have to do video motion tracking for interactive video campaign in my daily work.
If I’m used used to do that in Flash, I made a quick experiment to do the same in javascript.
Thanks to Olof Storm who made me a perfect corner pin motion tracking in After Effects, and I’ve been using some code from Steven Wittens to draw an image in perspective.
Click here to see the motion tracking demo (give it a bit of time to fully load).
What I’m doing in this demo is drawing a video in a canvas, and using the corner pin tracking data to draw an image on top of the video.
I made 2 different videos, you can switch between them using the “switch video” button.
In the first video, I get the current video frame number by calculation (current time by video fps). And in the second video, I get the frame number using a different technique that is know to be quite accurate when we do it in Flash, I call that a binary counter.
The idea behind the binary counter is to export the video with some white and black rectangle that describe graphically the real frame number of the video. We’ve been using this technique in Flash for a while in the advertising industry, and Zeh Fernando talked about it last year.
You can see the binary counter on the right border of the video. The code gets a pixel in each bit of the binary counter (16 bits), find out if the bit is white or black to get a 0 or 1 and calculate a number out of it.
While you might not see a difference between the two frame number calculations at first glance, seeking the video with the binary counter version (using the top-right blue control) will keep the image and the video “synchronised”. Pretty cool.
This version is not perfect compared to what we do in Flash, some frames still seem to be a bit “out”. I also couldn’t find a way to solve “pixel snapping drawing” in a canvas, in other words, being able to draw an image on half a pixel.
I did really miss some powerful drawing method from the graphic API in Flash. I would love to find in javascript a “beginBitmapFill” method, apply matrix directly to a drawing method and draw triangles in an easier way to use this plane drawing technic we use in Flash.
*update*
I just found out that what I call a binary counter is actually a VITC, which I didn’t know! Interesting!
Comments
Have you looked at the CreateJS suite of canvas tools – http://createjs.com/ EaselJS might have what you are looking for…
Yes I saw Grant Skinner released his libraries, I haven’t tried them yet. Paul Neave also gave me good directions for the pixel snapping. I need to look into this.
Pingback: Augmented Reality comes to JavaScript | CreativeJS
This is an amazing example! thank you very much for putting it together. I’ve never seen anything like it before.
Hey my pleasure, glad you like it.
Pingback: plus-me.at GOOGLE | Der Video ist klasse! Ich freue mich schon auf die Applikationen die man aus so etwas…
This is great! Out of curiosity, what did you use to burn VITC into videos prepped for HTML5 video?
An after project that we drop in another, it generates a layer and draw the VITC for the whole video. I can’t really share it as I didn’t make it and the author works in another company.
Cool, Can i tracking any object in any video ?
How to know tracking data ?
With after after effects for example:
https://helpx.adobe.com/after-effects/using/tracking-stabilizing-motion-cs5.html#motion_tracking_overview_and_resources
Search for motion tracking, then you export the data and use it in a programming language that can drive a video.
Thank you for your response, can i use opencv or other video lib to export the tracking data ?
For example :
1: i got video
2: select i want to track object
3: export tracking data
Js can do this ?
You understand that tracking an object as I’ve done is a manual task right? Something you do in after effect like in this video:
https://www.youtube.com/watch?v=sqsfLxtFQQ8
If you are trying to dynamically get this data, you will fall in domain like “motion capture” or “augmented reality”. More info there:
http://creativejs.com/2012/03/augmented-reality/
I suppose you can check what you can with javascript and opencv (node.js + opencv):
https://github.com/peterbraden/node-opencv
But this is still server side processing.
Unless you build a server that stream back a video that has been processed with openCV in real time, I don’t think there anything to do in the client. But I could be wrong, there’s so many attempts, experiments and solutions out there for processing video in real time.
I used AE and your video (tracking.mp4) try to export tracking data , but i still can not got this . i got the data not match your provide. could your tell me how to export data like your provide it ?
BTW i success to use opencv to got a tracking data , but still not match the video play speed .
My video fps is 25.00 . can i add skype or email ?
I had a script to export the data from AE formatted to be used straight away, just string manipulation. If there is something wrong using this data, you probably made a mistake somewhere.
This is a “corner pin” data set, and it has been done manually from after effects, you can’t get the exact same data.
If the speed doesn’t, wither the data is wrong, or you made a mistake using it (JS calculation, FPS mistake).
Big Thanks Romuald , I Solved my problem .. I use opencv to track object and export tracking data. Thank for your help !!!