Synchronising audio and video from multiple sources

[Dave2002]

A sudden burst of enthusiasm or at least interest in video has led me to some "market research". At a rough guess, Panasonic** seems to lead the field for camcorders, and at the bottom end there are some affordable models, though they do lack some useful features.

At the upper end, there's this - which is about to be released - http://www.amazon.co.uk/Panasonic-HC...onic+camcorder

This claims to be able to take input from other sources (e.g smartphones) at the same time. That could make things a lot more complicated!

**However it may be that there are models from JVC and Sony and others which are also good - but I've just not had time to check everything.

[Gordon]

The FZ300 that I mentioned has a 25-600mm (equivalent) with a constant f2.8 aperture throughout the range (one of the reasons I bought it).The lens benefits from the Panasonic/Leica partnership.

The previous model - FZ200 - has the same lens, but lacks 4k video.

I am surprised that the lens can maintain f2.8 out to 600mm. But we must remember that even the HDTV pixel count is about 1MPixel which means resolution is not that great and so optically the lens need not be that good. Maybe that is why they don't offer 4k with it?

[Dave2002]

1 Mpixel? Surely it should be more than that for HDTV - though perhaps for 720p that's about right. That camera is primarily a still camera, though, so should have more pixels for regular shots.

[Stunsworth]

I am surprised that the lens can maintain f2.8 out to 600mm. But we must remember that even the HDTV pixel count is about 1MPixel which means resolution is not that great and so optically the lens need not be that good. Maybe that is why they don't offer 4k with it?

They do offer 4k with the camera - from memory there is a choice of 24 FPS or 25 FPS. It also has 1080p and can shoot 720p at 100 FPS for slow motion effects - i.e. recorded at 100 FPS, replayed at 25 FPS.

Its 'real' function is as a 12 MP still camera.

Remember that the 600mm is a 35mm _equivalent_ figure - i.e. the same angle of view as a 600mm lens on a full frame 35mm camera. The actual focal length will be a lot shorter as it isn't a full frame sensor.

[Dave2002]

Back to synchronising ....

I found this article, which refers to Adobe Premiere CS5. Embedded within it there is a video, which is worth watching - http://video.stackexchange.com/quest...emiere-pro-cs6 This refers to multi-cam editing, which is just about where I am now. In any case, it's quite an interesting video.

I have one section of video now with the audio and video more or less lined up, and there are two cameras. I would like to be able to swap between the two cameras, pretty much as in the CS5 Youtube - but this may be a step too far with the software I'm using. There may be more than one audio track, either extracted and gain adjusted from one of the video cameras, or from a separate audio recording, though I think I have now more or less mastered how to figure out the audio synchronisation, so choosing the best audio track is now not such a problem once the audio and video are synced.

For me, lining up the tracks is getting easier as I get more familiar with the software, but it is still somewhat tricky. As the tracks all have audio, I can play and switch between the audio quite easily by turning the audio on/off for each track. This does enable me to get everything more or less into the right ball park, and then there can be a final tweaking. It's not so easy to use audio waveforms as a guide unless this check on the audio tracks is done first. This appears to be because the audio track of the most stable video has clearly been done with compression, whereas the other audio tracks haven't. This makes identifying video by use of the audio much harder. Once the audio tracks are more or less lined up, it is then a matter of expanding the timeline by a (possibly ver) large factor, and then trying to get the attack on similar audio lined up - but the audio will look quite different for compressed audio, so this has to be done where there are sharp attacks. Finally, the check is to play the lined up clips and switch between the audio (and video if wanted) to see that they are lined up.

Having got this far, I now need to check out software with multi-cam, which might make this editing a lot easier. I could take out an Adobe subscription, though I'm trying to avoid anything that expensive. I suspect that the software I'm using (Elements 14) may not have anything which would enable this to be done so easily.

[Dave2002]

Here is a follow up video showing all this might work with consumer level editing packages (e.g Elements 14) - https://www.youtube.com/watch?v=f_vRTbLn4lk This also shows how to do audio synchronisation, though it's quite interesting to note that visually at least there is a slight error in the "synchronised" audio. This is presumably because it's almost impossible to synchronise audio within any frame, and most playback equipment is not capable of shifting audio relative to video within frames. As mentioned earlier, this perhaps doesn't matter too much as the error will typically be at most 41.7 ms (that's for 24 fps - film rate), and less than that for higher frame rates - 25,30, 50, 60, 100 - assuming the audio synchronisation is to within one video frame. This Youtube video does show consumer level software being used - probably not as convenient at the more pro versions, but to avoid costlier solutions this might be worthwhile.

[Gordon]

They do offer 4k with the camera - from memory there is a choice of 24 FPS or 25 FPS. It also has 1080p and can shoot 720p at 100 FPS for slow motion effects - i.e. recorded at 100 FPS, replayed at 25 FPS.

Its 'real' function is as a 12 MP still camera.

Remember that the 600mm is a 35mm _equivalent_ figure - i.e. the same angle of view as a 600mm lens on a full frame 35mm camera. The actual focal length will be a lot shorter as it isn't a full frame sensor.

Agreed re still 12MP camera aspect, one needs the resolution there and if it can offer 4k the pixel size is small. That in itself is a challenge because each pixel gets that much less light.

My main fear re performance was the large [presumably all optical] zoom range - 25-600, 24:1 - not so much the image size cf 35mm. Its sensor is 6.2 x 4.6mm, an AR of 1.35:1 close to the 35mm ratio [24x18, 1.33...] which is a way off 16:9 usual in HDTV but its pixel size is such that getting a patch of 1920x1080 say is easy. At a large aperture there is also the greater need for geometry and colour [all glass surfaces refract] aberration correction with so much more glass surface in use - element number and area. A large f number means only axial areas of the glass elements are in use.

However you look at though it's an impressive performance.

[Dave2002]

,

Although it 's "obvious" that 4k may cause problems, and perhaps force consumers to buy larger storage, new cameras, and more expensive cameras, in my recent video editing project I can see that 4K or even higher resolution could have helped a lot, though would have brought problems with computer resources. The reason why very high resolution cameras could be useful, particularly for video, is that it would make the possibility of post production close ups, much higher quality.

Professional production teams may use more people, more (and better) equipment in order to obtain more usable video material for post production editing. Also, the experience they have would reduce, or avoid, some of the elephant traps which amateurs may fall into. Look at TV video (e.g. The Night Manager) and consider carefully how the shots have been filmed. Quite often the shots are really quite close up, with a modicum of panning to follow the actors - for example the scene around where Roper's son is kidnapped. Perhaps this was done actively during the filming, but it might also have been possible to do very similar sequences by filming slightly wider frame shots, and then doing the panning and maybe close up zooming in post production stages. Optical zooming can give even better results, but does require more hands on and experience during the filming.

Trying to do post production panning and zooming even on today's standards of HD material is very likely to show up the deficiencies. For example, video a choir and then try to do post production zoom in order to pick out a soloist. It is very easy to see that a zoom in to achieve a close up effect of (say) a factor or 2 or 3 - or more, could reduced the effective picture quality down to lower than today's SD video. Also, if there were any serious camera shake, that would also be exacerbated, though there are digital ways of overcoming that - though in consumer level software these are not always at the level of theoretical perfection.

4k video cameras may help amateur cinematographers to achieve better results in the future, and some may find the cost of the cameras, and the associated computer equipment and software, worth the expense. In any case we can still expect costs to drop for improved equipment finding its way into consumer markets. I can see also that some software packages, e.g Adobe Premiere and Final Cut Pro are now able to handle 4k clips. i would expect the trend to continue, and in a few years we may actually be going to another level of resolution above 4k.

I previewed the fairly short video I've been putting together of a performance of a violin concerto on my computer screen last night. At full screen resolution the quality is poor, so it would also be for TV or DVD. However, reduce the window size to about half and the video becomes acceptable - some might even find it good. All of the material used to put the video together was shot in either 720 or 1080 HD. This is not good enough for the kind of work represented by this project. 4k or better - bring it on!

[Dave2002]

Further investigation makes me wonder if tools such as Final Cut Pro X would be worth having. I have Adobe Premiere Elements, which was affordable, and I have worked out ways of doing multi-cam and audio syncing, but it's a bit fiddly. The method I use is similar to, but not the same as, the one in this video - which is based on iMovie - https://www.youtube.com/watch?v=ueh-PQT6pNg

I can also do essentially the same thing using the various versions of iMovie which I have on my machines. iMovie turns out to be more powerful than I'd thought, but it requires some effort to find out how to use it, and where the most useful features are. I suspect that many users who use these tools (if at all) for small projects (e.g holiday sequences) will not discover them.

If the audio is totally misaligned to video, this can be fairly obvious - providing the audio levels on the tracks are matched. If the audio is very slightly misaligned, then this may simply give a slight, but barely perceptible echo effect, which could be acceptable. It may also be that the audio from one source is significantly better (on the whole) than from some other sources, so it does make sense to try to get this right.

One other edit I did recently - two pianists playing on one piano - out of devilment I put a clip into the video which wasn't synchronised at all, and turned the sound down. Providing that both pianists are present and doing something - then providing the clip isn't long, then hardly anyone will notice that the hands on the keyboard aren't playing the right notes! This isn't a particularly good, or generally useful, thing to do, but it does show what can be "gotten away with" - providing enough visual clues and context are provided, and the audio track is good and solid. Glitches in the audio tracks are almost immediately noticeable. Obviously this rather dodgy trick won't work if totally different visual material is inserted.

So, back to Mulit-Cam and Final Cut Pro X. This video shows how to do it - and also shows that a tool such as FCPX is capable of doing all the sync operations
automatically - https://www.youtube.com/watch?v=kmFmyXWaQHM

There's some detail stuff from Apple here - https://support.apple.com/kb/PH12656?locale=en_US

https://support.apple.com/kb/PH17988?locale=en_US&viewlocale=en_US

https://support.apple.com/kb/PH17916?locale=en_US&viewlocale=en_US

https://support.apple.com/kb/PH12657?locale=en_US&viewlocale=en_US

https://support.apple.com/kb/PH17937?locale=en_US&viewlocale=en_US

https://support.apple.com/kb/PH12659?locale=en_US&viewlocale=en_US

https://support.apple.com/kb/PH17919?locale=en_US&viewlocale=en_US

I'm thinking of getting FCPX anyway, but it seems there is also a 30 days free trial - see http://www.apple.com/uk/final-cut-pro/whats-new/ which I guess I should try first.

It is certainly possible to do this kind of thing in Adobe Premiere CC - but that has a subscription model which at present I'm unwilling to buy into.

[Dave2002]

A very specific thing about making a post production video of a concert is that, during the performance part at least, times are fixed. This means that if a clip is taken from another camera, it has to match exactly into the main sequence, and to fit the audio track exactly - or at least as well as possible if obvious glitches are to be avoided.

This is specific to this kind of event, where times are fixed and all of the "footage" is to be shown essentially in real time.

Where there is applause, or there are gaps in the "live" event, it is possible to remove clips, or shorten or extend the clip timings, so the video editor has some (significant) control.

Other aspects of a recorded video could be different. For example, any introductory dialogue or interviews, could have timings adjusted, and different film editing techniques could be used, including three point editing, and removing gaps, and applause, but during the actual performance the timings have to be as accurate as possible and the editing options appear mostly to be to change the camera or camera angle, to pan and to zoom, and to cut between cameras.
Some video editors might highlight the audio for solos too - that is adjust the sound level (typically up) when it is clear that there is a solo playing, though this might not be appropriate for a classical concert. It probably still happens, though.

[Gordon]

They do offer 4k with the camera - from memory there is a choice of 24 FPS or 25 FPS. It also has 1080p and can shoot 720p at 100 FPS for slow motion effects - i.e. recorded at 100 FPS, replayed at 25 FPS.

Its 'real' function is as a 12 MP still camera.

Remember that the 600mm is a 35mm _equivalent_ figure - i.e. the same angle of view as a 600mm lens on a full frame 35mm camera. The actual focal length will be a lot shorter as it isn't a full frame sensor.

I saw a model 200 in a store the other day and the lens itself was marked with f no 2.8-4.5.

[Anastasius]

I could be wrong but I'm pretty sure that professional production companies don't muck about with zooming in the digital domain post-production.

[Dave2002]

I could be wrong but I'm pretty sure that professional production companies don't muck about with zooming in the digital domain post-production.

You're probably not wrong - for the most part - well maybe sometimes ..... PPCs will use experienced cameramen, good equipment, probably a script or several, and a lot of pre-planning. Also, if you think about it, with film and also analogue tape equpiment, doing any form of zoom or pan would have been difficult, though not impossible, after shooting. In the case of film, that might have required playing the film - possibly in slow motion, or even one frame at a time, and making an intermediate film to crop away the edges of each frame within a suitable window boundary. That would have reduced the final quality, though with good film the quality loss might have been acceptable to achieve the desired artistic effect. Mostly it would have been cheaper to ensure that the correct effects were recorded in the first place, but that is perhaps no longer always the case. Using analogue tape based systems would also have presented problems - and in any case most analogue recordings wouldn't have been good enough to allow a significant part of each frame to be cut away to give a good or even satisfactory result. Most analogue tape systems were surely used for TV production, rather than blockbuster film production, so had inherent quality limitations, and probably also short turn round times, so not enough time for significant post production editing with the systems available at the time.

However, with high resolution digital recording it should be possible to do such pan and zoom operations after the recording. Whether many companies do this I don't know - but a lot of what goes on in video production companies is related to costs. If it's possible to do these effects digitally in post production, and it becomes cheaper than doing things in other, more traditional ways, then it'll probably happen.

A great deal happens nowadays due to CGI and other techniques. Some older/more traditional directors are still wedded to older methods.

I saw a video about Clint Eastwood making a few films - probably Letters from Iwa Jima and the similar/related film Flags of our Fathers (both very good - particularly Letters - in which it was mentioned that he didn't use more modern techniques, and that perhaps added months and a lot of $$$$s on to the budget. The techniques he used were sound and good (and not by any means completely outdated) but were just not up to the level which is now possible with CGI modelling, and motion capture techniques.

I think the world has now changed, even for TV, with production companies using any number of digital effects and processes to get the results they think they need. I believe it is more flexible, and probably also a lot cheaper - given the standard of output they are now achieving. Just look at the opening of TV news programmes and try to figure out if the techniques demonstrated would have been possible decades ago. Companies such as Snell and Wilcox were doing special effects using analogue systems, but I'm sure that digital systems will, for the most part, be much more effective and cheaper.