Deep Learning-Accelerated Interactive Isosurface Visualization in Memory-Limited Settings via Speculative Raycasting
Keywords:
Isosurface visualization, speculative raycasting, deep learning acceleration, memory-limited environmentsAbstract
Interactive isosurface visualization plays a crucial role in exploring volumetric datasets, yet achieving real-time performance in memory-constrained environments remains a challenge. This paper presents a novel framework that integrates deep learning with speculative raycasting to accelerate isosurface rendering in such settings. By predicting likely ray traversal paths and surface intersections using a lightweight neural network, our approach reduces redundant computations and memory usage while maintaining high visual fidelity. Experimental evaluations on standard volumetric datasets demonstrate substantial improvements in rendering speed and responsiveness, even on devices with limited memory resources. This method enables practical deployment of high-quality interactive visualizations on edge devices and lightweight platforms.
References
J. Amanatides and A. Woo, “A fast voxel traversal algorithm for ray tracing,” in Proc. EG 1987- Tech. Papers, 1987, pp. 3–10.
M. B. Rodríguez, “State-of-the-art in compressed GPU-based direct volume rendering: State-of-the-art in compressed GPU-Based DVR,” Comput. Graph. Forum, vol. 33, pp. 77–100, 2014.
D. Bauer, Q. Wu, and K.-L. Ma, “FoVolNet: Fast volume rendering using foveated deep neural networks,” IEEE Trans. Vis. Comput. Graph., vol. 29, no. 1, pp. 515–525, Jan.2023.
J. Beyer, M. Hadwiger, and H. Pfister, “State-of-the-art in GPU-Based large-scale volume visualization,” Comput. Graph. Forum, vol. 34, pp. 13–37, 2015.
P. Cignoni, C. Montani, E. Puppo, and R. Scopigno, “Optimal isosurface extraction from irregular volume data,” in Proc. IEEE Symp. Volume Visual., 1996, pp. 31–38.
M. Ciżnicki, M. Kierzynka, K. Kurowski, B. Ludwiczak, K. Napierała, and J. Palczyński, “Efficient isosurface extraction using marching tetrahedra and histogram pyramids on multiple GPUs,” in Proc. Parallel Process. Appl. Math., 2012, pp. 343–352.
C. Crassin, F. Neyret, S. Lefebvre, and E. Eisemann, “GigaVoxels: Ray-guided streaming for efficient and detailed voxel rendering,” in Proc. Symp. Interactive 3D Graph. Games, 2009, pp. 15–22.
C. Dyken, G. Ziegler, C. Theobalt, and H.-P. Seidel, “High-speed marching cubes using HistoPyramids,” Comput. Graph. Forum, vol. 27, pp. 2028–2039, 2008.
L. Dyken, P. Poudel, W. Usher, S. Petruzza, J. Y. Chen, and S. Kumar, “GraphWaGu: GPU powered large scale graph layout computation and rendering for the web,” in Proc. Eurographics Symp. Parallel Graph. Visual., 2022, pp. 73–83.
T. Dykes, A. Hassan, C. Gheller, D. Croton, and M. Krokos, “Interactive 3D visualization for theoretical virtual observatories,” Monthly Notices Roy. Astronomical Soc., vol. 477, pp. 1495–1507, 2018.
K. Engel, “CERA-TVR: A framework for interactive high-quality teravoxel volume visualization on standard PCs,” in Proc. IEEE Symp. Large Data Anal. Visual., 2011, pp. 123–124.
T. Fogal, A. Schiewe, and J. Krüger, “An analysis of scalable GPU-based ray-guided volume rendering,” in Proc. IEEE Symp. Large-Scale Data Anal. Visual., 2013, pp. 43–51.
N. Fout, H. Akiba, K.-L. Ma, A. E. Lefohn, and J. Kniss, “High-quality rendering of compressed volume data formats,” in Proc. Eurographics/IEEE VGTC Symp. Visual., 2005, pp. 77–84.
N. Fout and K.-L. Ma, “Transform coding for hardware-accelerated volume rendering,” IEEE Trans. Vis. Comput. Graph., vol. 13, no. 6, pp. 1600–1607, Nov./Dec.2007.
E. Gobbetti, J. A. Iglesias Guitián, and F. Marton, “COVRA: A compression-domain output-sensitive volume rendering architecture based on a sparse representation of voxel blocks,” Comput. Graph. Forum, vol. 31, pp. 1315–1324, 2012.
M. Hadwiger, A. K. Al-Awami, J. Beyer, M. Agus, and H. Pfister, “SparseLeap: Efficient empty space skipping for large-scale volume rendering,” IEEE Trans. Vis. Comput. Graph., vol. 24, no. 1, pp. 974–983, Jan.2018.
M. Hadwiger, J. Beyer, W.-K. Jeong, and H. Pfister, “Interactive volume exploration of petascale microscopy data streams using a visualization-driven virtual memory approach,” IEEE Trans. Vis. Comput. Graph., vol. 18, no. 12, pp. 2285–2294, Dec.2012.
M. Hadwiger, C. Sigg, H. Scharsach, K. Bühler, and M. Gross, “Real-time ray-casting and advanced shading of discrete isosurfaces,” Comput. Graph. Forum, vol. 24, pp. 303–312, 2005.
J. Hasselgren, J. Munkberg, M. Salvi, A. Patney, and A. Lefohn, “Neural temporal adaptive sampling and denoising,” Comput. Graph. Forum, vol. 39, pp. 147–155, 2020.
L. Herzberger, “Residency octree: A hybrid approach for scalable web-based multi-volume rendering,” IEEE Trans. Vis. Comput. Graph., vol. 30, no. 1, pp. 1380–1390, Jan.2024.
D. Hoang, “Efficient and flexible hierarchical data layouts for a unified encoding of scalar field precision and resolution,” IEEE Trans. Vis. Comput. Graph., vol. 27, no. 2, pp. 603–613, Feb.2021.
A. Horé and D. Ziou, “Image quality metrics: PSNR vs. SSIM,” in Proc. 20th Int. Conf. Pattern Recognit., 2010, pp. 2366–2369.
H. Jacinto, R. Kéchichian, M. Desvignes, R. Prost, and S. Valette, “A web interface for 3D visualization and interactive segmentation of medical images,” in Proc. 17th Int. Conf. 3D Web Technol., 2012, pp. 51–58.
S. Jourdain, U. Ayachit, and B. Geveci, “ParaViewWeb: A web framework for 3D visualization and data processing,” Int. J. Comput. Inf. Syst. Ind. Manage. Appl., vol. 3, pp. 870–877, 2011.
A. S. Kaplanyan, A. Sochenov, T. Leimkühler, M. Okunev, T. Goodall, and G. Rufo, “DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos,” ACM Trans. Graph., vol. 38, pp. 1–13, 2019.
A. Kreskowski, G. Rendle, and B. Froehlich, “Efficient direct isosurface rasterization of scalar volumes,” Comput. Graph. Forum, vol. 41, pp. 215–226, 2022.
J. K. Li and K.-L. Ma, “P4: Portable parallel processing pipelines for interactive information visualization,” IEEE Trans. Vis. Comput. Graph., vol. 26, no. 3, pp. 1548–1561, Mar.2020.
P. Lindstrom, “Fixed-rate compressed floating-point arrays,” IEEE Trans. Vis. Comput. Graph., vol. 20, no. 12, pp. 2674–2683, Dec.2014.
B. Liu, G. J. Clapworthy, F. Dong, and E. Wu, “Parallel marching blocks: A practical isosurfacing algorithm for large data on many-core architectures,” Comput. Graph. Forum, vol. 35, pp. 211–220, 2016.
Y. Livnat and C. Hansen, “View dependent isosurface extraction,” in Proc. Vis., 1998, pp. 175–180.
Y. Livnat, H.-W. Shen, and C. R. Johnson, “A near optimal isosurface extraction algorithm using the span space,” IEEE Trans. Vis. Comput. Graph., vol. 2, no. 1, pp. 73–84, Mar.1996.
W. E. Lorenson and H. E. Cline, “Marching cubes: A high resolution 3D surface construction algorithm,” in Proc. 14th Annu. Conf. Comput. Graph. Interactive Techn., 1987, pp. 163–169.
G. Marmitt, A. Kleer, I. Wald, H. Friedrich, and P. Slusallek, “Fast and accurate ray-voxel intersection techniques for ISO-surface ray tracing,” in Proc. 9th Int. Fall Workshop Vis. Model. Visual., 2004, pp. 429–435.
S. Martin, H.-W. Shen, and P. McCormick, “Load-balanced isosurfacing on Multi-GPU clusters,” in Proc. Eurographics Symp. Parallel Graph. Visual., 2010, pp. 91–100.
J. Mensmann, T. Ropinski, and K. Hinrichs, “A GPU-Supported lossless compression scheme for rendering time-varying volume data,” in Proc. 8th IEEE/EG Int. Conf. Volume Graph., 2010, pp. 109–116.
M. M. Mobeen and L. Feng, “High-performance volume rendering on the ubiquitous WebGL platform,” in Proc. IEEE 14th Int. Conf. High Perform. Comput. Commun. IEEE 9th Int. Conf. Embedded Softw. Syst., 2012, pp. 381–388.
A. Mustafa, A. Mikhailiuk, D.-A. Iliescu, V. Babbar, and R. K. Mantiuk, “Training a task-specific image reconstruction loss,” in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis., 2021, pp. 2319–2328.
S. Parker, P. Shirley, Y. Livnat, C. Hansen, and P.-P. Sloan, “Interactive ray tracing for isosurface rendering,” in Proc. Visual., 1998, pp. 233–238.
A. Paszke, “PyTorch: An imperative style, high-performance deep learning library,” in Proc. 33rd Int. Conf. Neural Inf. Process. Syst., 2019, pp. 8026–8037.
F. Perez and B. E. Granger, “IPython: A system for interactive scientific computing,” Comput. Sci. Eng., vol. 9, pp. 21–29, 2007.
M. Raji, A. Hota, T. Hobson, and J. Huang, “Scientific visualization as a microservice,” IEEE Trans. Vis. Comput. Graph., vol. 26, no. 4, pp. 1760–1774, Apr.2020.
M. Raji, A. Hota, and J. Huang, “Scalable web-embedded volume rendering,” in Proc. IEEE 7th Symp. Large Data Anal. Visual., 2017, pp. 45–54.
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Proc. Med. Image Comput. Comput.-Assisted Interv., 2015, pp. 234–241.
J. Schneider and R. Westermann, “Compression domain volume rendering,” in Proc. IEEE Vis. Conf., 2003, pp. 293–300.
W. Schroeder, R. Maynard, and B. Geveci, “Flying edges: A high-performance scalable isocontouring algorithm,” in Proc. IEEE 5th Symp. Large Data Anal. Visual., 2015, pp. 33–44.
M. Schütz, “Potree: Rendering large point clouds in web browsers,” PhD thesis, Vienna Univ. of Technol., Vienna, Austria, 2016.
M. Schütz, B. Kerbl, and M. Wimmer, “Software rasterization of 2 billion points in real time,” Proc. ACM Comput. Graph. Interactive Techn., vol. 5, pp. 1–17, 2022.
P. Seetharaman, G. Wichern, B. Pardo, and J. Le Roux, “AutoClip: Adaptive gradient clipping for source separation networks,” in Proc. IEEE 30th Int. Workshop Mach. Learn. Signal Process., 2020, pp. 1–6.
T. Sherif, N. Kassis, M.-Ã. Rousseau, R. Adalat, and A. C. Evans, “BrainBrowser: Distributed, web-based neurological data visualization,” Front. Neuroinform., vol. 8, 2015, Art. no. 121755.
D. Smilkov, “TensorFlow.js: Machine learning for the web and beyond,” 2019, arXiv: 1901.05350.
S. Stegmaier, M. Strengert, T. Klein, and T. Ertl, “A simple and flexible volume rendering framework for graphics-hardware-based raycasting,” in Proc. 4th Int. Workshop Volume Graph., 2005, pp. 187–241.
S. K. Suter, “Interactive multiscale tensor reconstruction for multiresolution volume visualization,” IEEE Trans. Vis. Comput. Graph., vol. 17, no. 12, pp. 2135–2143, Dec.2011.
M. M. Thomas, K. Vaidyanathan, G. Liktor, and A. G. Forbes, “A reduced-precision network for image reconstruction,” ACM Trans. Graph., vol. 39, pp. 1–12, 2020.
W. Usher and V. Pascucci, “Interactive visualization of terascale data in the browser: Fact or fiction?,” in Proc. IEEE 10th Symp. Large Data Anal. Visual., 2020, pp. 27–36.
I. Wald, H. Friedrich, G. Marmitt, P. Slusallek, and H.-P. Seidel, “Faster isosurface ray tracing using implicit KD-trees,” IEEE Trans. Vis. Comput. Graph., vol. 11, no. 5, pp. 562–572, Sep./Oct.2005.
C. Wang, H. Yu, and K.-L. Ma, “Application-driven compression for visualizing large-scale time-varying data,” IEEE Comput. Graph. Appl., vol. 30, no. 1, pp. 59–69, Jan./Feb.2010.
Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr.2004.
S. Weiss, M. Chu, N. Thuerey, and R. Westermann, “Volumetric isosurface rendering with deep learning-based super-resolution,” IEEE Trans. Vis. Comput. Graph., vol. 27, no. 6, pp. 3064–3078, Jun.2021.
S. Weiss, P. Hermüller, and R. Westermann, “Fast neural representations for direct volume rendering,” Comput. Graph. Forum, vol. 41, pp. 196–211, 2022.
S. Weiss, M. IşIk, J. Thies, and R. Westermann, “Learning adaptive sampling and reconstruction for volume visualization,” IEEE Trans. Vis. Comput. Graph., vol. 28, no. 7, pp. 2654–2667, Jul.2022.
L. Wright, “Ranger - a synergistic optimizer,” 2019. [Online]. Available: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
Q. Wu, D. Bauer, M. J. Doyle, and K.-L. Ma, “Interactive volume visualization via multi-resolution hash encoding based neural representation,” IEEE Trans. Vis. Comput. Graph., vol. 30, no. 8, pp. 5404–5418, Aug.2024 doi: 10.1109/TVCG.2023.3293121.
H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comput. Imag., vol. 3, no. 1, pp. 47–57, Mar.2017.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Prof. Linda M. Faulkner, Dr. Chenghao Liu
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain full copyright of their published work. By submitting to an ERDAST journal and upon acceptance of the article, the author(s) agree to grant the journal a non-exclusive license to publish, reproduce, distribute, and archive the article. All articles are published under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0).
Under this license: Others may copy, distribute, display, remix, adapt, and build upon the work, even commercially, As long as proper credit is given to the original author(s) and source, A link to the license is provided, an, Any changes are clearly indicated.
???? License link: https://creativecommons.org/licenses/by/4.0/
