Feb
02
2010
Using Adobe Speech Search Cue Points in a Brightcove Player
Posted by: Bob de Wit in ActionScript, Brightcove, Flex
Adobe Speech Search is a function available in Adobe Premiere and SoundBooth CS4 that allows you to automatically transcribe spoken text in a video clip. In the screen shot above, you can see the metadata text transcription window on the right of the video loaded into Premiere.
The metadata can be incorporated into the video file as cue points, or can be exported to an XML file. The latter gives us some interesting possibilities for playback and navigation within a Brightcove Player.
You can extract the cue point information from an F4V or FLV created with Premiere by using SoundBooth or your own extraction tool based on the Flash media API. If you use SoundBooth, the extracted XML format looks something like this:
<CuePoint> <Time>44619</Time> <Type>event</Type> <Name>house</Name> <Parameters> <Parameter> <Name>source</Name> <Value>transcription</Value> </Parameter> <Parameter> <Name>duration</Name> <Value>409</Value> </Parameter> <Parameter> <Name>confidence</Name> <Value>36</Value> </Parameter> </Parameters> </CuePoint> <CuePoint>
Once we know this, it is easy to build a simple search and index plugin for BEML:
Sample code:
<?xml version="1.0" encoding="utf-8"?> <mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" width="300" height="412" backgroundColor="0xFFFFFF" xmlns:local="*"> <mx:Script> <![CDATA[ import mx.collections.ArrayCollection; import mx.controls.Alert; import flash.display.Stage; import com.brightcove.api.*; import com.brightcove.api.components.*; import com.brightcove.api.dtos.*; import com.brightcove.api.events.*; import com.brightcove.api.modules.*; import com.brightcove.api.search.*; import com.brightcove.api.utils.*; [Bindable] private var Results:ArrayCollection = new ArrayCollection(); private var Speech:XML; //Speech XML URL Root private var SpeechURL:String = "http://active6.com/blog/code/beml_speech/xml/"; //Placeholder for BC Player Reference private var Wrapper:BrightcoveModuleWrapper = null; //Placeholder for VideoPlayer Reference private var VideoPlayer:VideoPlayerModule = null; //Placeholder for Experience Reference private var Experience:ExperienceModule = null; public function setInterface(player:IEventDispatcher):void { //Save the passed BC player Object reference Wrapper = new BrightcoveModuleWrapper(player); Experience = Wrapper.getModule(APIModules.EXPERIENCE) as ExperienceModule; if (Experience == null) { Wrapper.addEventListener(ExperienceEvent.MODULES_LOADED, onModulesLoaded); Wrapper.loadModules(); } else { checkReady(); } } private function onModulesLoaded(event:ExperienceEvent):void { Wrapper.removeEventListener(ExperienceEvent.MODULES_LOADED, onModulesLoaded); Experience = Wrapper.getModule(APIModules.EXPERIENCE) as ExperienceModule; if (Experience != null) { checkReady(); } } private function checkReady():void { if (Experience.getReady()) { InitApp(); } else { Experience.addEventListener(ExperienceEvent.TEMPLATE_READY, onTemplateReady); } } private function onTemplateReady(event:ExperienceEvent):void { Experience.removeEventListener(ExperienceEvent.TEMPLATE_READY, onTemplateReady); InitApp(); } private function InitApp():void { VideoPlayer = Wrapper.getModule(APIModules.VIDEO_PLAYER) as VideoPlayerModule; VideoPlayer.addEventListener(MediaEvent.CHANGE, onMediaChange) GetSpeechFile(); } private function onMediaChange(evt:Object):void { GetSpeechFile(); } private function GetSpeechFile():void { try { var video:Object = VideoPlayer.getCurrentVideo(); var request:URLRequest = new URLRequest(SpeechURL + video.id + ".xml"); request.method = URLRequestMethod.GET; var loader:URLLoader = new URLLoader(); loader.dataFormat = URLLoaderDataFormat.TEXT; loader.addEventListener(Event.COMPLETE, handleResults); loader.addEventListener(IOErrorEvent.IO_ERROR, ioErrorHandler); loader.load(request); } catch( e:Error) { } } private function handleResults(evt:Event):void { var response:String = evt.target.data as String; Speech = new XML(response); } private function ioErrorHandler(evt:Event ):void { } private function SearchClick():void { Results.removeAll(); for each( var cuepoint:XML in Speech.CuePoint ) { if( cuepoint.Name == SpeechSearch.text ) { var o:Object = new Object(); o.Time = cuepoint.Time; o.Label = SpeechSearch.text + " : Time Index " + cuepoint.Time; Results.addItem(o); } } } private function SkipClick():void { var o:Object = ResultList.selectedItem; var index:Number = new Number(o.Time)/1000; Status.text = "Skipping to index " + index.toString(); VideoPlayer.seek(index); } ]]> </mx:Script> <mx:VBox x="0" y="0" width="100%" height="100%"> <mx:HBox width="100%" height="26" verticalAlign="middle"> <mx:Label text="Search"/> <mx:TextInput width="100%" id="SpeechSearch" text="house" backgroundColor="#D6D6D6"/> <mx:Button label="Go" width="49" id="ButtonSearch" click="SearchClick()"/> </mx:HBox> <mx:List dataProvider="{Results}" id="ResultList" labelField="Label" borderStyle="none" click="SkipClick()" width="100%" height="100%"></mx:List> <mx:Label id="Status" text="Search a term and click in the results" /> </mx:VBox> </mx:Application>
Entries (RSS)