# **4D Ray-Space and Ultra-Wide Area FTV**

**Masayuki Tanimoto Hirokuni Kurokawa**  Nagoya Industrial Science Research Institute University of Aizu Nagoya, Japan Aizu Wakamatsu, Japan tanimoto@nagoya-u.jp aiguo@u-aizu.ac.jp

Fig. 1 Definition of orthogonal ray-space.

*z*

*y*

Reference Plane

*z* = 0

*x*

Ray crossing the reference plane

Intensity *f*

ϕθ),,,(

*yxf*

Intersection *yx* ),(

Direction ϕθ),(

Fig. 2 Concept of "a group of rays through one point".

Fig. 3 Ray capture, view generation and ray synthesis by using the concept of "a group of rays through one point".

(1) and (2) give the 4D orthogonal ray-space (*x, y, θ, φ*) of a group of rays through one point. By changing *θ* and *φ*, this group of rays forms a plane in (*x, y, θ, φ*) space. To visualize it, (*x, y, θ*)

*x* and *y* of a ray with direction (*θ, φ*) through a point P(X, Y, Z) are given by

space with fixed *φ* and (*x, y, φ*) space with fixed *θ* are considered in the following.

*x=X* − *tanθZ* (1)

(2)

*tanφ cosθ*

*y=Y* +

**Abstract –Ultra-wide area FTV is an FTV with very wide viewing zone where motion parallax is realized. 4D orthogonal ray-space is analyzed and applied to ultra-wide area FTV. Ray-space of "a group of rays through one point" is derived in 4D orthogonal ray-space. It is extended to obtain ray-space captured by linear arrangement cameras. View generation of ultra-wide area FTV needs rays that are not captured by real cameras. These rays are synthesized by interpolating the captured ray-space so that the intersections of the captured ray-space and the ray-space of rays emitted from a light source have the same color.**

## **1 INTRODUCTION**

FTV (Free-viewpoint Television) [1]-[8] is visual media that transmits all ray information of a 3D space and enables immersive 3D viewing as if we were actually there. Omnidirectional FTV is an FTV with very wide field of view (FOV), that is 360-degree FOV. Ultra-wide area FTV is an FTV with very wide viewing zone that gives motion parallax. FTV was developed based on ray-space representation [9]-[12]. 4D ray-space is needed for immersive 3D viewing with full parallax.

We proposed two types of ray-space, orthogonal ray-space and spherical ray-space. Orthogonal ray-space is used for linear and planar camera arrangements and spherical ray-space is used for circular and spherical camera arrangements. 4D spherical ray-space was analyzed and applied to omnidirectional FTV [13]. In this paper, 4D orthogonal ray-space is analyzed and applied to ultra-wide area FTV. Ray capture and view generation in 4D orthogonal ray-space are presented.

## **2 4D ORTHOGONAL RAY-SPACE**

## **2.1 4D Orthogonal Ray-Space of Rays through One Point**

Fig. 1 shows the definition of orthogonal ray-space. As shown in Fig. 1, one ray is expressed by intersection (*x, y*) and direction (*θ, φ*) on a reference plane. Therefore, one ray is expressed by one point in 4D parameter space (*x, y, θ, φ*). This parameter space is 4D orthogonal ray-space. Each point in the 4D ray-space has an intensity *f* of the ray. Therefore, *f* is expressed as *f* (*x, y, θ, φ*).

 Here, "a group of rays through one point" is considered as shown in Fig. 2. This concept is used for ray capture, view generation and ray synthesis as shown in Fig. 3. Fig. 3(a) shows ray capture by a real camera, and view generation by a virtual camera collecting rays through one point. Fig. 3(b) shows ray synthesis. If the surface of an object is a diffusion surface, one point on the surface emits rays with equal magnitude. Therefore, if one ray is known, other rays can be synthesized.

Fig. 2 Concept of "a group of rays through one point".

Fig. 3 Ray capture, view generation and ray synthesis by using the concept of "a group of rays through one point".

*x* and *y* of a ray with direction (*θ, φ*) through a point P(X, Y, Z) are given by

$$\begin{aligned} \chi &= X - \tan\theta Z \\ \chi &= Y + \frac{\tan\phi}{\cos\theta} Z \end{aligned} \qquad (1)$$

(1) and (2) give the 4D orthogonal ray-space (*x, y, θ, φ*) of a group of rays through one point. By changing *θ* and *φ*, this group of rays forms a plane in (*x, y, θ, φ*) space. To visualize it, (*x, y, θ*) space with fixed *φ* and (*x, y, φ*) space with fixed *θ* are considered in the following.

From (1) and (2), the following equations are derived.

$$1 + \left(\frac{\varkappa - X}{Z}\right)^2\tag{3}$$

$$1 + (\tan \theta)^2 = \left(\frac{\varkappa - Y}{Z \tan \phi}\right)^2\tag{4}$$

For fixed *φ,* (1), (3) and (4) give the top view, front view and side view of (*x, y, θ*) space, respectively. The top view is a straight line with slope -*Z* and both the front and side views are hyperbolic curves in (*x, y, tanθ*) space. Therefore, a group of rays through one point forms a hyperbolic curve in (*x, y, tanθ*) space with fixed *φ* as shown in Fig. 4. Three examples of hyperbolic curves for *φ*>0, *φ***=**0 and *φ*<0 are shown.

**2.2 4D Orthogonal Ray-Space Captured by Linear Arrangement Cameras** 

*θ* (6)

shown in Fig. 6, *x* and *y* are given by

*y= <sup>Z</sup>*<sup>c</sup>

*y=Z*c*tanφ* 1 + *tan*<sup>2</sup>

and

*y= <sup>Z</sup>*<sup>c</sup>

respectively.

*x= X*<sup>c</sup> − *tanθZ*<sup>c</sup> (5)

(5) and (6) give ray-space captured by linear arrangement cameras.

expressions of the hyperbolic cylinder and the plane are given by

*cos<sup>θ</sup> tanφ* for any (8)

*θ* for any (7)

*cos<sup>θ</sup> tanφ=Zctan<sup>φ</sup>* <sup>1</sup> <sup>+</sup> *tan<sup>2</sup>*

Let arrange many cameras on a line that is set on the *xz*-plane (*y*=0) and parallel to the *x*-axis. When an omnidirectional camera at P(*X*c, 0, *Z*c) on the line captures a ray with direction (*θ, φ*) as

Fig. 6 Ray capture by linear arrangement cameras

By varying *X*c, (5) and (6) form a hyperbolic cylinder in (*x, y, tanθ*) space with fixed *φ* as shown in Fig. 7. They form a plane in (*x, y, tanφ*) space with fixed *θ* as shown in Fig. 8. The

For fixed *θ,* (1) and (2) form a straight line in (*x, y, tanφ*) space as shown in Fig. 5. Three examples of straight lines for *θ*>0, *θ***=**0 and *θ*<0 are shown.

Fig. 4 Rays through a point P(X, Y, Z) form a hyperbolic curve in (*x, y, tanθ*) space with fixed *φ*.

Fig. 5 Rays through a point P(X, Y, Z) form a straight line in (*x, y, tanφ*) space with fixed *θ*.

#### **2.2 4D Orthogonal Ray-Space Captured by Linear Arrangement Cameras**

Let arrange many cameras on a line that is set on the *xz*-plane (*y*=0) and parallel to the *x*-axis. When an omnidirectional camera at P(*X*c, 0, *Z*c) on the line captures a ray with direction (*θ, φ*) as shown in Fig. 6, *x* and *y* are given by

$$\alpha = X\_{\c} - \tan \theta Z\_{\c} \tag{5}$$

$$y = \frac{Z\_{\c}}{\cos \theta} \tan \phi = Z\_{\c} \tan \phi \sqrt{1 + \tan^{2} \theta} \tag{6}$$

(5) and (6) give ray-space captured by linear arrangement cameras.

Fig. 6 Ray capture by linear arrangement cameras

By varying *X*c, (5) and (6) form a hyperbolic cylinder in (*x, y, tanθ*) space with fixed *φ* as shown in Fig. 7. They form a plane in (*x, y, tanφ*) space with fixed *θ* as shown in Fig. 8. The expressions of the hyperbolic cylinder and the plane are given by

$$\mathbf{y} = \mathbf{Z}\_c \tan \phi \sqrt{\mathbf{1} + \tan^2 \theta} \qquad \text{for any } \mathbf{x} \qquad (7)$$

and

$$\mathbf{y} = \frac{\mathbf{z}\_{\ast}}{\cos \theta} \tan \phi \qquad \text{for any } \mathbf{x} \tag{8}$$

respectively.

Fig. 7 Ray-space in (*x, y, tanθ*) space with fixed *φ* captured by linear arrangement cameras.

Fig. 9 Ray capture and synthesis for ultra-wide area FTV.

space and their intersection in (*x, y, φ*) space with fixed *θ*. The intersection is given by

*<sup>Z</sup>*c*-Z* ) (7)

*Z*c*-Z*

 *, Ycos<sup>θ</sup>*

*Z*c*-Z*

(*x, y, tanφ*) = (*X - Ztanθ, <sup>Z</sup>*c*<sup>Y</sup>*

It is on a horizontal plane of *y = <sup>Z</sup>*c*<sup>Y</sup>*

color. This color is given to the ray (*x*0*, y*0*, θ*0*, φ*0).

Some of emitted rays are captured by the real cameras. These rays are given by the intersection of emitted and captured ray-spaces. Fig 10 shows captured ray-space, emitted ray-

Many intersections are obtained by varying *θ*. Fig. 11 shows the trajectory of this intersection.

Fig. 10 Captured ray-space, emitted ray-space and their intersection in (*x, y, φ*) space with fixed *θ*.

 *.* When *Z* is correct depth, the intersections have the same

Fig. 8 Ray-space in (*x, y, tanφ*) space with fixed *θ* captured by linear arrangement cameras.

# **3 VIEW SYNTHESIS FOR ULTRA-WIDE AREA FTV**

4D orthogonal ray-space is applied to ultra-wide area FTV [11]. Ray capture and synthesis for ultra-wide area FTV is shown in Fig. 7. Rays are captured by many cameras on a line (*y=*0*, z=Z*c) and virtual camera views are synthesized from the captured rays. Let one of the rays captured by a virtual camera be (*x*0*, y*0*, θ*0*, φ*0). This ray is synthesized because it is not captured by real cameras. When *z* of the light source of this ray is assumed to be *Z*, the position (*X, Y, Z*) of the light source is determined. This light source emits rays in various directions. Let ray-space of the emitted rays be (*x, y, θ, φ*).

Fig. 9 Ray capture and synthesis for ultra-wide area FTV.

Some of emitted rays are captured by the real cameras. These rays are given by the intersection of emitted and captured ray-spaces. Fig 10 shows captured ray-space, emitted rayspace and their intersection in (*x, y, φ*) space with fixed *θ*. The intersection is given by

$$(\text{x, y, } \tan \phi) = (X \text{ - Ztan}\theta, \frac{Z\_\ast Y}{Z\_\ast \cdot Z}, \frac{Y \cos \theta}{Z\_\ast \cdot Z}) \tag{7}$$

Many intersections are obtained by varying *θ*. Fig. 11 shows the trajectory of this intersection. It is on a horizontal plane of *y = <sup>Z</sup>*c*<sup>Y</sup> Z*c*-Z .* When *Z* is correct depth, the intersections have the same color. This color is given to the ray (*x*0*, y*0*, θ*0*, φ*0).

Fig. 10 Captured ray-space, emitted ray-space and their intersection in (*x, y, φ*) space with fixed *θ*.

**4 CONCLUSION**

the same color.

**References**

**ACKNOWLEDGEMENTS**

Special Session 5, December 2004.

(April 2012). (invited paper)

(online 24 February 2012). (invited paper)

10.1017/ATSIP.2012.5 (invited paper)

Ultra-wide area FTV with full parallax is realized by 4D orthogonal ray-space processing. Ray-space of "a group of rays through one point" is derived in 4D orthogonal ray-space. It is used for ray capture, view generation and ray synthesis. Ray-space of "a group of rays through one point" forms a hyperbolic curve in (*x, y, tanθ*) space with fixed *φ* and a straight line in (*x, y, tanφ*) space with fixed *θ.* By varying the camera position along a line, ray-space captured by linear arrangement cameras is obtained. This ray-space forms a hyperbolic cylinder in (*x, y, tanθ*)

Rays for view generation of ultra-wide area FTV are synthesized by interpolating the captured ray-space using the concept of "a group of rays through one point". It is done so that the intersections of the captured ray-space and the ray-space of rays emitted from a light source have

[1] Masayuki Tanimoto, "Free Viewpoint Television," The Journal of Three Dimensional

[2] Masayuki Tanimoto, "Free Viewpoint Television - FTV", Picture Coding Symposium 2004,

[3] Masayuki Tanimoto, "Overview of Free Viewpoint Television," Signal Processing: Image

[4] Masayuki Tanimoto, Mehrdad Panahpour Tehrani, Toshiaki Fujii, Tomohiro Yendo, "Free-Viewpoint TV", IEEE Signal Processing Magazine, vol.28, no.1, pp.67-76, January 2011. [5] Masayuki Tanimoto, Mehrdad Panahpour Tehrani, Toshiaki Fujii, Tomohiro Yendo, "FTV for 3-D Spatial Communication", Proceedings of the IEEE, Vol. 100, No. 4, pp. 905-917

[6] Masayuki Tanimoto, "FTV: Free-viewpoint Television", Signal Processing:Image Communication, Vol. 27, Issue 6, pp. 555-570 (June 2012). doi:10.1016/j.image.2012.02.016

[7] Masayuki Tanimoto, "FTV (Free-viewpoint Television)", APSIPA Transactions on Signal and Information Processing, Vol. 1, Issue 1, e4 (14 pages) (August 2012). doi:

[8] Masayuki Tanimoto, "FTV (Free-viewpoint Television) and its International Standardization", IEIE Transactions on Smart Processing and Computing, vol. 6, no. 6,

[9] T. Fujii, "A Basic Study on Integrated 3-D Visual Communication", Ph.D dissertation in

[10] T. Fujii, T. Kimoto, M. Tanimoto, "Ray Space Coding for 3D Visual Communication",

[11] M. Tanimoto, A. Nakanishi, T. Fujii, T. Kimoto, "The Hierarchical Ray-Space for Scalable 3-D Image Coding", Picture Coding Symposium 2001, pp.81-84, April 2001. [12] Toshiaki Fujii, Masayuki Tanimoto, "Free-Viewpoint TV System Based on Ray-Space

[13] Masayuki Tanimoto and Hirokuni Kurokawa, "Omnidirectional FTV", Proc. of 2017 International Conference on 3D Immersion, Brussels, Belgium, December 12, 2017.

pp.415-427, December 2017. doi:10.5573/IEIESPC.2017.6.6.415 (Invited paper)

Representation", SPIE ITCom 2002, Vol. 4864-22, pp.175-189, Aug. 2002.

space with fixed *φ* and a plane in (*x, y, tanφ*) space with fixed *θ*.

This research is supported by JSPS KAKENHI Grant Number 16H04370.

Images, vol.15, no.3, pp.17-22, September 2001 (in Japanese).

Communication, vol. 21, no. 6,pp. 454-461,July 2006.

engineering, The University of Tokyo, 1994 (in Japanese).

Picture Coding Symposium '96, pp. 447-451, Mar. 1996.

Fig. 11 Trajectory of the intersection of captured ray-space and emitted ray-space in (*x, y, φ*) space by varying *θ*.

The process of this ray synthesis is summarized in the flowchart of Fig. 12.

Fig. 12 Flowchart of ray synthesis.

#### **4 CONCLUSION**

Ultra-wide area FTV with full parallax is realized by 4D orthogonal ray-space processing. Ray-space of "a group of rays through one point" is derived in 4D orthogonal ray-space. It is used for ray capture, view generation and ray synthesis. Ray-space of "a group of rays through one point" forms a hyperbolic curve in (*x, y, tanθ*) space with fixed *φ* and a straight line in (*x, y, tanφ*) space with fixed *θ.* By varying the camera position along a line, ray-space captured by linear arrangement cameras is obtained. This ray-space forms a hyperbolic cylinder in (*x, y, tanθ*) space with fixed *φ* and a plane in (*x, y, tanφ*) space with fixed *θ*.

Rays for view generation of ultra-wide area FTV are synthesized by interpolating the captured ray-space using the concept of "a group of rays through one point". It is done so that the intersections of the captured ray-space and the ray-space of rays emitted from a light source have the same color.

## **ACKNOWLEDGEMENTS**

This research is supported by JSPS KAKENHI Grant Number 16H04370.

#### **References**

