DepthGate for Glasses free video: Advanced Multiview Generation

Displays creating the glasses-free 3D effect are called “autostereoscopic” – AS3D. Such displays require video content in special formats at the input.
The most wide-spread AS3D formats: 2D+Depth and some types of the “multiview” (5-view, 8-view, 9-view). Triaxes DepthGate is a software intended for creating video content in such formats.The source data for the program is video content  in the stereo 3D format – 2 channels that are usually used for 3D TV and by cinemas showing 3D movies with glasses. DepthGate can convert traditional 3D (with glasses) into glasses-free 3D.

Let’s see how this conversion is performed and pay attention to some peculiarities allowing to get better results. We took a scenic film about Peterhof’s fountains as source data (side-by-side format, 1920×1080 res). Here’s one of the shots from the movie (Fig. 1).

Fig.1. Shot from source movie in side-by-side format (stereo pair).

Because this is a real stereo pair, it contains the information about the scene depth. DepthGate can retrieve this information – calculate the depth map. To calculate and see the depth map we need to select the shot we are interested in, and go to the Depth Map tab. The program will automatically calculate the depth map for the shot.

This example – the shot with water splashes – is taken on purpose. This video example is difficult to be shown in the 2D+Depth format, because the situation is tricky: we need to make a choice – which distance to set for transparent water splashes through which we can see the background. Should it be the distance to the water splashes or the distance to the background? This choice is not easy even for a human, and it is far more difficult for an automatic algorithm.

Fig.2. Going to Depth map tab and getting stereo pair.

The “Depth” slider enables to control the transparency – it is convenient in terms of demonstration as the depth map corresponds to the source image (Fig. 3) This control element was made to simplify the demonstration on the screen, it doesn’t influence the export result.
It can be seen that the program separated the foreground objects (water splashes) from the background (grassy slope).

Fig.3 Transparency adjustment when Depth map overlays source frame.

The 2D+Z video is enough for some AS3D displays. Even now we can export the video in this format. (Fig.4)

Fig.4. Shot in 2D+Depth format. This video format is compatible with 3D displays from Philips, Dimenco and some other models.

To export content in the 2D+Depth format we open the Export dialogue and set the parameters (Fig.5).

Fig.5 Export dialogue. Choosing 2D+Depth format.

The Depth information is used by the display to calculate some views (lines of sight) for the image, while the microlens layer of the screen creates the 3D effect. As it was noted before, this example is quite a difficult one as there is a great deal of water splashes and semi-transparent objects, however, this method allows to show the 3D effect on the screen.

In order to model this effect on a conventional (2D) screen we generate the views for this scene and use the animation effect – consecutive demonstration of frames (Fig.6).

Fig.6. Animated demonstration of frames created on the basis of 2D+Depth.

Some models of AS3D displays (Philips, Dimenco, etc.) are compatible with video in the 2D+Depth format. Calculation of views – lines of sight – is performed by displays’ instrumentary. Other display models, for example Realcel, Alioscopy, etc., receive video in the multiview format at the input. DepthGate software can make calculations and prepare video in such a format.

To make calculations we go to the Multiview tab. (Fig.7) The program automatically makes calculations for the given number of frames (9,8 or 5) and shows the resulting frames in the mosaic mode.

Fig.7. Calculations and demonstration of 9 frames in mosaic mode.

The resulting multiview frames can be seen in more details, frame by frame, in the “Select Frame” mode or in the frame animation mode (Fig.8).

Fig.8. Multiview frame demonstration in animation mode.

Finally, we’ve come to the most interesting stage. It wasn’t by chance that we chose a difficult shot. The purpose was to show how DepthGate copes with this task. Indeed, as the farness from the viewer can’t be set unambiguously when the Depth for transparent objects is defined, it leads to the background being distorted in the areas where it is overlaid by semi-transparent objects.

For example, in Fig.8 we see the changes in contours of the people and objects on the hill slope that can be seen through water splashes. (Fig. 8 – area between arrows). In many cases DepthGate allows to solve the problem. For that the frame generation algorithm, that takes into account 2 source frames of the stereo pair, is used. Fig.9 shows parameters that you need to set in order to use the 2-frame generation method.

Fig9 Two generation methods. On the left – 2D+Depth method, on the right – 2-frame mode.

Fig.10 shows comparison of 2 generation methods.

Fig.10 Two generation methods. On the left – using 1 frame and depth map. On the right – using 2 source frames.

 Similar cases with semi-transparent objects can take place when video contains glass, reflection, water, smoke, etc. Let’s have a look on one more example where the 2-frame generation method is used. Fig.11 shows a series created using 1 frame and the depth map (2D+Depth). Fig.12 shows a series created using 2 frames.

Fig.11 2D+Depth generation – pay attention to the way the contours of the wheel and parts of the caterpillar track on the background change.

Fig.12 2-frame generation (2D+Depth). The wheels and the back caterpillar track seen through the dust look more true to life.

Translucency. Fig. 13 shows a stereopair with a dragonfly and depth maps calculated by DepthGate for each frame . The translucent wings of the insect is a special case for conversion into multi-view 3D. Figure 14 shows how DepthGate copes with the task – here’s a 9-view series rendered out of two source stereo frames and depth maps. Fig. 15 shows the animation of the same series.

Fig. 13. Stereopair with a dragonfly and depth maps

Fig. 14. 9-view series rendered out of two source stereo frames and depth maps

Fig. 15. Animation: 9-view series rendered out of 2-frames.

Conclusion

2-frame generation allows to get a more true to life image in many cases.
1-frame-and-depth-map generation gives an opportunity to change parallax settings (intensity of the 3D effect) in a greater way.
Depending on the scene type the user can choose the generation method to get a better effect.
For each scene one can choose different generation settings.
As a result of the video conversion, one can get the best possible result for each scene.