NYU Tandon cuts a rug with new 3D video technology | NYU Tandon School of Engineering
Summary
The NYU Tandon teams project will address those obstacles, reducing bandwidth consumption and delivery latency, and increasing power consumption efficiency so that PCVs can be streamed far more easily.With recent advances in the key enabling technologies, we are now at the verge of completing the puzzle of teleporting holograms of real-world humans, creatures and objects through the global Internet, said Yong Liu, professor of electrical and computer engineering (ECE), who leads the research team that includes ECE Professor Yao Wang and R. Luke DuBois, associate professor in the Technology, Culture and Society Department.The success of the proposed research will contribute towards wide deployment of high quality and robust PCV streaming systems that facilitate immersive augmented, virtual and mixed reality experience and create new opportunities in many domains, including education, business, healthcare and entertainment, said Liu.As part of the project, DuBois is heading up a collaboration with the Mark Morris Dance Groups Dance Center a cultural hub in Brooklyn that serves 1,500 students and the Department of Dance at NYU Tisch School of the Arts, to test PCV streaming with human motion.Point-Cloud Video holds tremendous potential to transform a range of industries, and Im excited that the research team at NYU Tandon prioritized dance education to reap those benefits early, said Jelena Kovaevi, NYU Tandon Dean. Collaborating with dancers from NYU Tisch and Mark Morris makes the innovative power of engineering imminently real.Dancers from both organizations will perform on a volumetric capture stage a specialized production environment for shooting 3D video at NYU Tandon @ the Yard, the schools production and research facility for emerging media at the Brooklyn Navy Yard. Our project also exposes dancers to an understanding of the possibilities and challenges of 3D video, giving diverse students and practitioners insight into the ways in which computers, engineering and media positively impact their work and arouse their interest in computing.We are excited to embark on a dynamic collaboration with the exceptional team at NYU Tandon, and look forward to exploring the potential and innovation this technology promises to bring to the realms of dance education and creative expression, said Sarah Marcus, Director of Education and Community Engagement, Mark Morris Dance Group.From a technical perspective, beyond developing the PCV testbed used specifically for dance experimentation, the NYU Tandon researchers want the project to make fundamental strides in three areas that will advance PCV more broadly.First, they will develop efficient PCV compression using object-based coding schemes, hierarchical slicing structures, and advanced prediction techniques that enables bitstream adaptation based on the predicted viewers field of view and the network throughput. Lastly, they will design edge-based caching algorithms, multi-user delivery strategies, and cross-user Field-of-View (FoV) predictions to enhance PCV streaming efficiency and robustness.Mark Morris Dance Group and the Department of Dance at NYU Tisch will recruit students and instructors to take part in the project, and will start performing on the volumetric capture stage in Spring 2024. The project officially started on October 1, 2023.This project builds on NYU Tandons advanced work to build the next generation of foundational wireless communications and media processing technologies, as well as extensive industry collaborations with leaders in immersive video and virtual production, including Evercoast, Final Pixel, Argus, Epic Games, and Walt Disney, to bring research-grade emerging technology within reach of media, entertainment and cultural institutions while developing technologists who are exceptionally prepared to further the field of emerging media.