Synchronising audio and video from multiple sources

[Gordon]

Thanks Gordon. Both videos were recorded on digital equipment. One was a Sony video camera - I don't know the model, the other was a Zoom H2 Handycam

OK. Googling I did not find a “Zoom H2 Handycam” only an audio recorder called Zoom H2. “Handycam” appears to be a Sony model name for its camcorders. So it looks as if both video cameras were Sony?

The Sony was set up by someone else who has given me access to the material.

So you only have a compressed file on DVD or in your computer??

Perhaps foolishly I recorded some of the video material on the Zoom in HD. The good thing about that is that some of the images are very clear - though unfortunately my camera work wasn't very good. I think the Sony material was recorded in 720 HD, so not as high resolution as the Zoom.

So both videos are in some form of HD. In the US there are 2 permitted “HDTV” formats: 720I or 720P and 1080I or 1080P. The P means double the frame rate so that in fact 720P and 1080P are at a 60Hz frame rate. This doubles the byte rate and file size compared to the I formats which retain the standard 30Hz frame rate. The aspect ratio is 16:9 so the horizontal pixel density is 1280 for the 720 version and 1920 for the 1080. Mixing/editing the two requires an Up [720 to 1080] or Down [1080 to 720] conversion to get the frames the same size [your video card will do a conversion of its own for display – a flat LCD panel? - depending on its native pixel size and aspect ratio eg 1024x576 for 16:9] and refresh rate which may not be 30Hz but probably is. The frame rates are probably nominally the same - as long as they are both I or P format. Assuming the I format each frame is 2x1920x1080x30 bytes/second or 7.5GBytes/min. For 720I the numbers are: 2x1280x720x30 bytes/second or 3.3GBytes/min, all uncompressed. Compression by 100:1 or more requires a number of techniques and many include dropping pixel density [loss of spatial resolution] and also dropping frame rate [loss of movement rendition ie jerkiness] as well as the usual MPEG tricks. One of the ways to synchronise video with unlocked clocks is to drop frames when the Read is slower than Write or repeat a frame for vice versa. IF the clocks happen to be very close but not actually locked and the video material is short in duration then it’s possible to avoid losing or repeating frames but that is down to luck rather than design.

Post production zooming in to the Sony material, for example to pick out soloists, seems to rather quickly give poor results - at least in the edit preview. This is for several reasons. Firstly the angle of view used covers more of the performers, so zooming in on a single face or part of the scene exacerbates the quality issues compared to the Zoom material. The Zoom material seems in some sequences to be closer to the performers, so further zooming in does not degrade the images so much as with the other material. Secondly, the resolution of the Zoom recordings, being higher, does allow for much clearer close up shots. I rather think that all of the material was recorded at 30 fps, but I'd have to check. Modern digital kit can often record in a range of standards, aspect ratios, frame rates etc. I think most of the Zoom audio was in PCM, while the Sony audio was in Dolby AC3. The Zoom can also do mp4 (aac) audio at a number of different compression ratios.

No surprise that zooming degrades because it’s probably a digital zoom rather than an optical one; the former selects a subset of pixels to make a whole frame. The latter is always better and starting from a higher sample density is also better.

If the best bits of the Zoom material can be combined with the Sony material then the final result could be better. If I were doing this again I'd hope to have much better camera work! The Sony was on a static mount.

The separate audio tracks of most of the pieces were also recorded on a HiMD minidisc recorder, though in fact the audio format used was ATRAC rather linear PCM which had been an option. This was decided to give a longer running length, and also as the theoretical quality improvement of PCM over ATRAC did not seem to make a substantial difference for these recordings. I made the CD of the event months ago - didn't take me very long, though I did have to do some conversion using that old Sony Sonicstage program on a PC. One piece wasn't completely recorded on the MD recorder, so I used one of the video soundtracks for that - probably the Zoom PCM version.

Arguably the MD sound track is better than either of the video soundtracks, but it is rather debatable now whether it would be worth using that for the eventual final video. Since the video soundtracks are already tied to their respective video, and the experience of watching a video is different it is now undoubtedly easier just to work only with the video source material.

Seems to me you’d be best using the Sony locked off camera source as the basis of your project keeping its audio and [720I? ] video sync’d together. Then try and drop into that file some video frames of close ups from your own material in the same frame format as and when the music suggests eg solos etc zooming in on individuals. For this to look right you’ll need to get out of any compressed domain. You will also have to find a way for the editing software to identify the Drop-In and Out points [frame boundaries] for both the target file and the second source whose frames must over-write those of the target. Clearly the Drop In frames have to be from exactly -give or take a frame but not much more - the same place in the audio if they are selected for specific points in the music. Lip sync is visible if the audio is in advance of the video by 1 frame [40ms] and 2 frames if the opposite. A seasoned video technician will see a lip sync error of about 10ms. With music though you have a lot more leeway - as long as there are no singers in shot.

[MrGongGong]

I've not got time to read all of this in great detail
BUT given that you are effectively 'bodging' it together from various sources you don't have to have every single movie clip in perfect sync with the audio.
My strategy (as Gordon says above) would be to choose which source the audio is coming from, lay that down with the movie and then insert the best of the other movie materials, if you more or less line them up in the edit software you should be reasonably close.

All a far cry from when I was technician for a RR Bennet film music course in the 1980's, in the first showing the film was 16mm and I sat in the hall with the edited audio on a B77 with my hand on the varispeed adjusting as we went in an attempt to hit the significant sync points.....a bit more hit and miss but probably much more fun than staring at a computer screen.

[Dave2002]

Sorry - I didn't have the Zoom gadget or its manual in front of me when I put up the last post,

This is, I think, the model - http://www.amazon.co.uk/Zoom-Q2HD-Ha...video+recorder Q2HD. obviously I got the model number wrong.

That now seems to be rather pricey.

I think the reference to Handycam is also probably wrong. I picked up my video camera a few days ago - a tape based machine from years back, and I think that word etched itself into my brain. The other video camera used was definitely a Sony.

[MrGongGong]

That now seems to be rather pricey.

Amazon often just make up the prices of things (in a DM style ? )

http://www.maplin.co.uk/p/zoom-q3hd-full-hd-24-inch-handy-video-recorder-used-b32nl

http://www.maplin.co.uk/p/zoom-q2hd-handy-video-recorder-n99nj

[Dave2002]

Amazon often just make up the prices of things (in a DM style ? )

http://www.maplin.co.uk/p/zoom-q3hd-full-hd-24-inch-handy-video-recorder-used-b32nl

http://www.maplin.co.uk/p/zoom-q2hd-...recorder-n99nj

Although I've not been a great fan of that Q2HD recorder, it's probably not a bad buy at the Maplin price of £99.99.
One thing I never did get round to testing was whether it would work with a standard USB PSU. I used batteries, which tend to run out very quickly, and I was reluctant to try just any PSU. I also wondered whether a mains powered USB PSU would inject significant noise, which is another reason for sticking with batteries. I didn't quite get round to testing how long the batteries would last in different modes. I have a hunch that the highest quality audio and video modes drain the batteries faster. Sometimes it ran down in around 30 minutes - a pain for a long piece, or a long session.

At the Maplin price (your second link) I might be tempted to buy my own model of that. Is there anything else remotely comparable at the price point - round £100? On the other hand it might be better to spend two or three times (or even more) that amount to get a more robust and more up to date higher quality unit from other manufacturers.

Specs and info on the Q2HD are here - https://www.zoom-na.com/products/fie...video-recorder

On the face of it rather good, but the unit seems just too small and has a somewhat awkward and plasticky - maybe flimsy - feel about it. Nevertheless, at a low price ..... !!???

[Dave2002]

Seems to me you’d be best using the Sony locked off camera source as the basis of your project keeping its audio and [720I? ] video sync’d together. Then try and drop into that file some video frames of close ups from your own material in the same frame format as and when the music suggests eg solos etc zooming in on individuals. For this to look right you’ll need to get out of any compressed domain. You will also have to find a way for the editing software to identify the Drop-In and Out points [frame boundaries] for both the target file and the second source whose frames must over-write those of the target. Clearly the Drop In frames have to be from exactly -give or take a frame but not much more - the same place in the audio if they are selected for specific points in the music. Lip sync is visible if the audio is in advance of the video by 1 frame [40ms] and 2 frames if the opposite. A seasoned video technician will see a lip sync error of about 10ms. With music though you have a lot more leeway - as long as there are no singers in shot.

I am homing in on this, but it's taking time. I agree in principle with using the Sony material as a base on which to put the other material.

One arguably interesting thing to note is how we perceive musicians in concerts. The video (see below) I've recently been working with has a violin soloist perhaps 30 feet from my viewing position. That would imply an around (slightly less than) 30 ms delay of the audio behind the video (the microphones were also around the same distance away from the performers - though I had considered putting microhones closer to them), but we may adjust for this because it is natural for us to do so. In a large concert hall there could easily be 0.1 sec delay of the audio behind the video, but mostly we don't notice.

I have been practising synchronising audio to video with a slightly different section of the concert, which features a violin concerto. I can now get things roughly right, but looking at the violinist's fingers during rapid passages does suggest there is still some error. In the live situation the violinist was perhaps 30 feet away, so visually that does suggest that we wouldn't notice a 30 ms lag in sound compared with the visual impression, and in a large concert hall we do presumably get used to lags of 100 ms (0.1 sec) or more. It's also possible that our brains compensate for the timing discrepancy between the visual and aural stimuli in different environments. [Sorry for the repetition following insertion of a new piece of text ....!]

I'd really like to mnimise the perceived timing errors. I don't know if this is possible with the package I'm currently using - Adobe Premiere Elements. Even if I can get the audio and video synced to within 1 frame, there is still the possility of more than 10 ms error - which presumably is normally adjusted for by putting delays in the audio channel, or if the video is slightly ahead of the audio, then delaying the video by a few frames, then putting a more precise delay in the audio channel to bring the audio and video back into perceptual sync. Although I'm not trying to do lip sync adjustment, presumably the same techniques could be used for watching violinists. Again, what I don't know is whether any such adjustment can be made withing the editing software, or whether lip sync is something which can only be done during replay.

I found this type of question raised by others - https://forums.adobe.com/thread/737923?tstart=0 - and a few sensible answers.

I note the comment that a one frame error between the video and audio is generally not noticeable, but a 2 frame error nearly always is.
If it's really not possible to adjust the audio, then presumably it should be possible to put in padding in small increments e.g 5ms, 10ms,20ms at the start of the audio track using a separate audio editor (e.g audacity) then resync on the start of the new audio track. That would be a bit of a kludge - also a faff, but should make it possible to get closer.

Also the comments seem to suggest that the audio and video should be treated as separate clips and (re-)synced about every 5-10 minutes to compensate for any drift due to clock discrepancies between audio and video iputs. This does seem reasonable, if problems are detected, though hopefully that may not be needed too often.

One slight concern I have at present is that in Premiere Elements it is possible to stretch either or both audio and video tracks, which is the last thing one wants if trying to synchronise tracks which were made with reasonably high precision clocks. There may be a way of imposing a rigidity constraint to avoid this, though so far I've not found out for sure if this is really necessary etc. One page recommended linking tracks together once synchronised to reduce the risk of careless editing errors spoiling the synchronisation.

[Anastasius]

..... Nevertheless, at a low price ..... !!???

Remember the adage ......Buy Cheap - Buy Twice

[Dave2002]

Remember the adage ......Buy Cheap - Buy Twice

Indeed. Though sometimes that's not a bad idea - depends how cheap. The Zooms are serviceable enough, but only have digital zoom (which is a waste of space - why bother - can always do digital zoom later on?), and need to be mounted on a tripod to give good results - which is indeed the case for most video cameras. Arguably they are better at the audio side than some other video kit. The video frame rates are 30 fps - so for UK PAL frame converters might be needed. Software would do the job, though might introduce some unwanted artefacts.

I am quite taken with this suggested by mangerton - http://www.digitalversus.com/camcord...5325/test.html - but it is a lot more. Currently around or over £600, though does do 3D, with an adapter, if wanted. Does anyone have experience of 3D? Would that only work with Panasonic TVs?
Interesting that that one seems to have 5:1 microphones - though how they work on a small device such as this I can't figure. It does look as though at least an external stereo mic can be plugged in. Does appear to be very good - though at a price. Note that this model has a frame rate of 50 fps - which makes it UK PAL compatible, presumably without the need for any further frame rate or other conversion.

Also the still camera suggested by Steve with the 4k feature seems a good buy. http://news.panasonic.co.uk/pressrel...-in-4k-1192890
It might also be possible to buy one for not much more than £300 if one shops around. If the zoom lens keeps the f2.8 feature over the full range that would definitely make it something worth considering. Not sure if that one works with external microphones. This one also works in 4k at 30 fps, or 12 fps in photo mode at ISO 6400.

Doing 4k more seriously appears to be a much more expensive way to go, and there'd almost inevitably be further upgrades needed down the line - for post production computers etc. - more memory, faster processors - you name it.

[Stunsworth]

If the zoom lens keeps the f2.8 feature over the full range that would definitely make it something worth considering. Not sure if that one works with external microphones

Dave, there's a 3.5mm socket for an external microphone.

[Dave2002]

Remember the adage ......Buy Cheap - Buy Twice

Actually those "cheap" Zoom devices from Maplin now seem to have been phantoms, or maybe they were the very end of the retail stock. If anyone wants one they may now have to look to eBay.

[Gordon]

......The video frame rates are 30 fps - so for UK PAL frame converters might be needed. Software would do the job, though might introduce some unwanted artefacts.

Yes it will, especially if not done well. It is easy to do off line [quite hard in real time] in a computer by substituting 5 25Hz frames for every 6 input 30Hz ones. Easy to do by dropping a frame each tenth of a second although this produces some judder. To avoid judder you'd need to interpolate temporally and that needs lots of storage and quite clever memory management involving motion compensation which can get quite hairy eg in a zoom where everything is moving radially rather than H or V directions. Going the other way means ADDING a frame every tenth of a second and that's done by repetition [or by clever interpolation again]. But watch the playout issues - if the audio cannot be interrupted the new 5 frames MUST occupy the exact same time period as the original 6 and that means the playout clocks must be right. One can use audio processing software to remove the extra 30Hz frame time period.

Beware compatibility. PAL is an obsolete format now but its frame rate of 25 Hz is still used for digital but all TV lines are scanned in one sweep - so called Progressive Scan indicated by the P after the line number eg 1080P. In the analogue days a composite of 2 fields [odd lines first from the top then even lines also from the top] at a rate of twice the frame rate was used for all analogue formats including NTSC. This is Interlace and denoted by eg1080I. The clocks for P run at twice the rate of I.

Check that the camera really is 30Hz frame rate. The old US standard NTSC frame rate was 29.97Hz which is 1000/1001 times 30Hz. It's unlikely but you MAY need to adjust that 1 frame in 1000 as well, once every 30 seconds or so - clocks again.

Note that this model has a frame rate of 50 fps - which makes it UK PAL compatible, presumably without the need for any further frame rate or other conversion.

See above re Interlace and compatibility. A 50Hz FRAME rate is too fast for PAL analogue by a factor of 2. Alternate frames will have to be dropped [judder perhaps?] and the clocks changed to read out one set of odd lines in the first field and then the even ones in the next. Some modern TVs deal with both formats but need to know from the input digital file/signal what they are getting.

If the zoom lens keeps the f2.8 feature over the full range that would definitely make it something worth considering.

Unlikely - f2.8 is quite a wide aperture for any lens and particularly for a zoom. At the extreme end of the zoom range [long FL] the aperture will reduce in size by about a couple of stops thus reducing by about 4x the light reaching the sensor. The cheaper the lens the greater the loss of aperture at long FL.

[Stunsworth]

Unlikely - f2.8 is quite a wide aperture for any lens and particularly for a zoom. At the extreme end of the zoom range [long FL] the aperture will reduce in size by about a couple of stops thus reducing by about 4x the light reaching the sensor. The cheaper the lens the greater the loss of aperture at long FL.

The FZ300 that I mentioned has a 25-600mm (equivalent) with a constant f2.8 aperture throughout the range (one of the reasons I bought it).The lens benefits from the Panasonic/Leica partnership.

The previous model - FZ200 - has the same lens, but lacks 4k video.

YouTube has examples of video from both cameras.

[Dave2002]

Gordon

Thanks for the detailed reply. Sorry for the confusion re PAL, but I think (and you seem to agree) that 25 fps is used in the UK - though I think some TVs up this rate by a factor of 2 or 4. Conversion from 30 fps (or its near rough equivalent) is likely to introduce unwanted effects, which may (perhaps?) be more noticeable if the video quality is basically good. One of the best ways of doing frame rate conversion seems to be using Optical Flow techniques, as suggested by Michael Black and others.

I found a very interesting article on this, but the link isn't immediately to hand, so I'll post it later.

Essentially the method makes "virtual" frames between the real frames in the source video. Of course in formats such as MPEG many of the frames are themselves derived frames, using either forward or backward prediction from neighbouring frames. Optical flow methods insert even more frames in between, and then sample from those in order to effect the frame rate conversion. It may also be that some algorithms don't actually generate these intermediate "virtual" frames, unless there is a need to do so. This kind of method would probably be hard or impossible to do in real time, but with powerful/fast computers it is possible to do this kind of thing in offline mode in order to give better results when the material is distributed with the transformed frame rate.

[Dave2002]

The FZ300 that I mentioned has a 25-600mm (equivalent) with a constant f2.8 aperture throughout the range (one of the reasons I bought it).The lens benefits from the Panasonic/Leica partnership.

The previous model - FZ200 - has the same lens, but lacks 4k video.

YouTube has examples of video from both cameras.

Steve (and Gordon)

I do think it's possible that the Panasonic camera does a reasonable job at f2.8. There was another camera which had a terrific zoom from Fuji a few years ago, and I think it gave an equivalent to a 1000mm lens, but its light gathering performance dropped off at high zoom. The model was a Fujifilm HS EXR50 - probably now discontinued. Look for reviews, and see what the sample images look like. At first glance the pictures with the high zoom level do look good, but I believe that the aperture problems do/did let that model down.

From what I've heard, and my own experience of the Panasonic/Lumix cameras, it seems likely that this newer camera does solve many of the problems.
I am interested!

[Stunsworth]

Dave, if I manage to find anything interesting to snap I'll post some sample video on YouTube this weekend with a link here.

No promises mind you.