Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Depth Map Upscaling for Three-Dimensional Television: The Edge-Weighted Optimization Concept
Mid Sweden University, Faculty of Science, Technology and Media, Department of Information Technology and Media. (Realistic3D)ORCID iD: 0000-0002-2578-7896
2012 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

With the recent comeback of three-dimensional (3D) movies to the cinemas, there have been increasing efforts to spread the commercial success of 3D to new markets. The possibility of a 3D experience at home, such as three-dimensional television (3DTV), has generated a great deal of interest within the research and standardization community.

A central issue for 3DTV is the creation and representation of 3D content. Scene depth information plays a crucial role in all parts of the distribution chain from content capture via transmission to the actual 3D display. This depth information is transmitted in the form of depth maps and is accompanied by corresponding video frames, i.e. for Depth Image Based Rendering (DIBR) view synthesis. Nonetheless, scenarios do exist for which the original spatial resolutions of depth maps and video frames do not match, e.g. sensor driven depth capture or asymmetric 3D video coding. This resolution discrepancy is a problem, since DIBR requires accordance between the video frame and depth map. A considerable amount of research has been conducted into ways to match low-resolution depth maps to high resolution video frames. Many proposed solutions utilize corresponding texture information in the upscaling process, however they mostly fail to review this information for validity.

In the strive for better 3DTV quality, this thesis presents the Edge-Weighted Optimization Concept (EWOC), a novel texture-guided depth upscaling application that addresses the lack of information validation. EWOC uses edge information from video frames as guidance in the depth upscaling process and, additionally, confirms this information based on the original low resolution depth. Over the course of four publications, EWOC is applied in 3D content creation and distribution. Various guidance sources, such as different color spaces or texture pre-processing, are investigated. An alternative depth compression scheme, based on depth map upscaling, is proposed and extensions for increased visual quality and computational performance are presented in this thesis. EWOC was evaluated and compared with competing approaches, with the main focus was consistently on the visual quality of rendered 3D views. The results show an increase in both objective and subjective visual quality to state-of-the-art depth map upscaling methods. This quality gain motivates the choice of EWOC in applications affected by low resolution depth.

In the end, EWOC can improve 3D content generation and distribution, enhancing the 3D experience to boost the commercial success of 3DTV.

Place, publisher, year, edition, pages
Sundsvall, Sweden: Mittuniversitetet , 2012. , 57 p.
Series
Mid Sweden University licentiate thesis, ISSN 1652-8948 ; 92
Keyword [en]
3d video, 3DTV, video coding, capture, distribution, EWOC, depth map upscaling, time-of-flight
National Category
Signal Processing
Identifiers
URN: urn:nbn:se:miun:diva-17048ISBN: 978-91-87103-41-4 (print)OAI: oai:DiVA.org:miun-17048DiVA: diva2:561904
Presentation
2012-11-22, O111, Mittuniversitetet - Holmgatan 10, Sundsvall, 09:00 (English)
Opponent
Supervisors
Available from: 2012-10-22 Created: 2012-09-24 Last updated: 2017-08-22Bibliographically approved
List of papers
1. Depth Map Upscaling Through Edge Weighted Optimization
Open this publication in new window or tab >>Depth Map Upscaling Through Edge Weighted Optimization
2012 (English)In: Proceedings of SPIE - The International Society for Optical Engineering / [ed] Atilla M. Baskurt, Robert Sitnik, SPIE - International Society for Optical Engineering, 2012, Art. no. 829008- p.Conference paper, Published paper (Refereed)
Abstract [en]

Accurate depth maps are a pre-requisite in three-dimensional television, e.g. for high quality view synthesis, but this information is not always easily obtained. Depth information gained by correspondence matching from two or more views suffers from disocclusions and low-texturized regions, leading to erroneous depth maps. These errors can be avoided by using depth from dedicated range sensors, e.g. time-of-flight sensors. Because these sensors only have restricted resolution, the resulting depth data need to be adjusted to the resolution of the appropriate texture frame. Standard upscaling methods provide only limited quality results. This paper proposes a solution for upscaling low resolution depth data to match high resolution texture data. We introduce We introduce the Edge Weighted Optimization Concept (EWOC) for fusing low resolution depth maps with corresponding high resolution video frames by solving an overdetermined linear equation system. Similar to other approaches, we take information from the high resolution texture, but additionally validate this information with the low resolution depth to accentuate correlated data. Objective tests show an improvement in depth map quality in comparison to other upscaling approaches. This improvement is subjectively confirmed in the resulting view synthesis.

Place, publisher, year, edition, pages
SPIE - International Society for Optical Engineering, 2012
Keyword
3DTV, depth map, upscaling, time-of-flight, view synthesis, optimization, edge detection
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:miun:diva-15805 (URN)10.1117/12.903921 (DOI)000304302300007 ()2-s2.0-84861935064 (Scopus ID)STC (Local ID)978-081948937-1 (ISBN)STC (Archive number)STC (OAI)
Conference
3-Dimensional Image Processing (3DIP) and Applications II;Burlingame, CA;24 January 2012through26 January 2012;Code90039
Available from: 2012-02-16 Created: 2012-01-31 Last updated: 2017-08-22Bibliographically approved
2. Adaptive depth filtering for HEVC 3D video coding
Open this publication in new window or tab >>Adaptive depth filtering for HEVC 3D video coding
2012 (English)In: 2012 Picture Coding Symposium, PCS 2012, Proceedings, IEEE conference proceedings, 2012, 49-52 p.Conference paper, Published paper (Refereed)
Abstract [en]

Consumer interest in 3D television (3DTV) is growing steadily, but current available 3D displays still need additional eye-wear and suffer from the limitation of a single stereo view pair. So it can be assumed that auto-stereoscopic multiview displays are the next step in 3D-at-home entertainment, since these displays can utilize the Multiview Video plus Depth (MVD) format to synthesize numerous viewing angles from only a small set of given input views. This motivates efficient MVD compression as an important keystone for commercial success of 3DTV. In this paper we concentrate on the compression of depth information in an MVD scenario. There have been several publications suggesting depth down- and upsampling to increase coding efficiency. We follow this path, using our recently introduced Edge Weighted Optimization Concept (EWOC) for depth upscaling. EWOC uses edge information from the video frame in the upscaling process and allows the use of sparse, non-uniformly distributed depth values. We exploit this fact to expand the depth down-/upsampling idea with an adaptive low-pass filter, reducing high energy parts in the original depth map prior to subsampling and compression. Objective results show the viability of our approach for depth map compression with up-to-date High-Efficiency Video Coding (HEVC). For the same Y-PSNR in synthesized views we achieve up to 18.5% bit rate decrease compared to full-scale depth and around 10% compared to competing depth down-/upsampling solutions. These results were confirmed by a subjective quality assessment, showing a statistical significant preference for 87.5% of the test cases.

Place, publisher, year, edition, pages
IEEE conference proceedings, 2012
Keyword
3-D displays; 3-D television; 3D video coding; Auto stereoscopic; Bit rates; Coding efficiency; Consumer interests; Depth information; Depth Map; Depth value; Edge information; High energy; Multiview displays; Multiview video; Stereo view; Subjective quality assessments; Test case; Upsampling; Upscaling; Video frame; Viewing angle
National Category
Signal Processing
Identifiers
urn:nbn:se:miun:diva-16211 (URN)10.1109/PCS.2012.6213283 (DOI)000306962400013 ()2-s2.0-84864026988 (Scopus ID)STC (Local ID)978-1-4577-2048-2 (ISBN)STC (Archive number)STC (OAI)
Conference
29th Picture Coding Symposium, PCS 2012;Krakow;7 May 2012through9 May 2012;Code91163
Projects
Realistic3D
Available from: 2012-09-14 Created: 2012-05-14 Last updated: 2017-08-22Bibliographically approved
3. Improved edge detection for EWOC depth upscaling
Open this publication in new window or tab >>Improved edge detection for EWOC depth upscaling
2012 (English)In: 2012 19th International Conference on Systems, Signals and Image Processing, IWSSIP 2012, IEEE conference proceedings, 2012, 1-4 p.Conference paper, Published paper (Refereed)
Abstract [en]

The need for accurate depth information in three-dimen-sional television (3DTV) encourages the use of range sensors,i.e. time-of-flight (ToF) cameras. Since these sensors provideonly limited spatial resolution compared to modern high res-olution image sensors, upscaling methods are much needed.Typical depth upscaling algorithms fuse low resolution depthinformation with appropriate high resolution texture frames,taking advantage of the additional texture information in theupscaling process. We recently introduced a promising up-scaling method, utilizing edge information from the textureframe to upscale low resolution depthmaps. This paper exam-ines how a more thorough edge detection can be achieved byinvestigating different edge detection sources, such as inten-sity, color spaces and difference signals. Our findings showthat a combination of sources based on the perceptual quali-ties of the human visual system (HVS) leads to slightly im-proved results. On the other hand these improvements implya more complex edge detection.

Place, publisher, year, edition, pages
IEEE conference proceedings, 2012
Series
Systems, Signals and Image Processing (IWSSIP), ISSN 2157-8672 ; 19
Keyword
3DTV, EWOC, depth map, ToF, upscaling, perceptual edge detection, HVS, CIE2000
National Category
Signal Processing
Identifiers
urn:nbn:se:miun:diva-16210 (URN)2-s2.0-84863949324 (Scopus ID)STC (Local ID)978-1-4577-2191-5 (ISBN)STC (Archive number)STC (OAI)
Conference
2012 19th International Conference on Systems, Signals and Image Processing, IWSSIP 2012;Vienna;11 April 2012through13 April 2012;Category numberCFP1255E-ART;Code91138
Available from: 2012-09-14 Created: 2012-05-14 Last updated: 2017-08-22Bibliographically approved
4. Incremental depth upscaling using an edge weighted optimization concept
Open this publication in new window or tab >>Incremental depth upscaling using an edge weighted optimization concept
2012 (English)In: 3DTV-Conference, 2012, Art. no. 6365429- p.Conference paper, Published paper (Refereed)
Abstract [en]

Precise scene depth information is a pre-requisite in three-dimen-sional television (3DTV), e.g. for high quality view synthesis inautostereoscopic multiview displays. Unfortunately, this informa-tion is not easily obtained and often of limited quality. Dedicatedrangesensors, suchastime-of-flight(ToF)cameras, candeliverre-liable depth information where (stereo-)matching fails. Nonethe-less, since these sensors provide only restricted spatial resolution,sophisticated upscaling methods are sought-after, to match depthinformation to corresponding texture frames. Where traditionalupscaling fails, novel approaches have been proposed, utilizingadditional information from the texture for the depth upscalingprocess. We recently proposed the Edge Weighted OptimizationConcept (EWOC) for ToF upscaling, using texture edges for ac-curate depth boundaries. In this paper we propose an importantupdate to EWOC, dividing it into smaller incremental upscalingsteps. We predict two major improvements from this. Firstly, pro-cessing time should be decreased by dividing one big calculationinto several smaller steps. Secondly, we assume an increase inquality for the upscaled depth map, due to a more coherent edgedetection on the video frame. In our evaluations we can showthe desired effect on processing time, cutting down the calculationtime more than in half. We can also show an increase in visualquality, based on objective quality metrics, compared to the origi-nal implementation as well as competing proposals.

Keyword
3DTV, EWOC, DIBR, time-of-flight, depth map, upscaling, edge detection, incremental, optimization, view synthesis
National Category
Signal Processing
Identifiers
urn:nbn:se:miun:diva-17023 (URN)10.1109/3DTV.2012.6365429 (DOI)2-s2.0-84872059517 (Scopus ID)STC (Local ID)978-146734905-5 (ISBN)STC (Archive number)STC (OAI)
Conference
2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video, 3DTV-CON 2012;Zurich;15 October 2012through17 October 2012;Category numberCFP1255B-ART;Code94817
Projects
3D video: Capture and Compression for Distribution
Available from: 2012-09-25 Created: 2012-09-19 Last updated: 2017-08-22Bibliographically approved

Open Access in DiVA

fulltext(5980 kB)1206 downloads
File information
File name FULLTEXT02.pdfFile size 5980 kBChecksum SHA-512
83361a2ba8d948b83cdebb0451d890e6202eeffc54dea1184d3e5076a4661d8d33d124c7254feebab29ed56b3be2c580785062bfa8c18afe8075d3f876fa144e
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Schwarz, Sebastian
By organisation
Department of Information Technology and Media
Signal Processing

Search outside of DiVA

GoogleGoogle Scholar
Total: 1206 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1262 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf