Banner image: With a new tool called Augmented Physics, students can transform static diagrams in textbooks into moving simulations. (Credit: Gunturu et al., 2024, UIST '24)
With just an iPad, students in any classroom across the world could soon in any physics textbook鈥攖ransforming these static images into 3D simulations that run, leap or spin across the page.
These new, living textbooks are the brainchild of a team of computer scientists led by Ryo Suzuki at 兔子先生传媒文化作品.听
鈥淯sually, those diagrams are fixed. We have to imagine what happens,鈥� said Suzuki, assistant professor in the ATLAS Institute and Department of Computer Science. 鈥淏ut what if we could take any static diagram from any textbook and make it interactive?鈥�
He and his colleagues recently took home a 鈥渂est paper鈥� award for听their work at the 37th 听this October in Pittsburgh.听
Picture this: You crack open your high school physics textbook to a drawing of a skier teetering at the top of a jump. The diagram is supposed to teach you about topics like kinetic energy, such as the energy of a skier zooming down a hill. But the skier never actually moves. He just sits there.听
Until now.
For the new study, Suzuki鈥檚 team designed a tool that allows students to record that diagram with their iPads. Then, after users adjust some settings, the tool draws on artificial intelligence to get the skier to move across the screen. He zooms down the hill, hits the jump and soars. It鈥檚 scientifically accurate, too: Students can adjust the friction of the snow, making their skier zoom faster or slower with a few swipes.
Suzuki sees the project as a chance to reinvent the textbook鈥攇iving learners the opportunity to not just read about physics but see topics like momentum and electrical currents in action.
鈥淲e believe the future of education and learning should be more interactive and personalized,鈥� said Suzuki, who began the project as a researcher at the University of Calgary in Canada.
It鈥檚 alive!
Aditya Gunturu, first author of the study, remembers his own days learning about physics. He used to sit in science class in high school, thinking there must be a better way to visualize topics to convey motion.
鈥淚 was so frustrated,鈥� said Gunturu, a master鈥檚 student in computer science at the University of Calgary. 鈥淚 wanted to visualize these concepts.鈥�
Augmented Physics visualizes electric currents flowing through a circuit.听(Credit: Gunturu et al., 2024, UIST '24)
He鈥檚 not alone: Gunturu鈥檚 mother is a biology teacher in India and often draws on a blackboard to show students, for example, how cells divide鈥攁 phenomenon textbooks can only capture in snapshots.
The team's invention, Augmented Physics, could help.听
The tool relies on a model called Segment Anything from the tech company Meta. It鈥檚 a computer visualization tool that allows users to click on a photo to isolate particular objects鈥攁 dog, or maybe a face. Similarly, through Augmented Physics, students and teachers select various objects inside a diagram, such as the skier and the ski jump, and assign those objects roles. The AI then applies some basic physics, such as the force of gravity, to make those objects move.
It works for a several different kinds of diagrams, too. They include optics diagrams鈥攕o students can simulate how a prism refracts sunlight, for example鈥攁nd electrical circuits鈥攕o they can see currents zip along wires. In a full circle moment, the researchers tested their tool out on some of those same diagrams in Gunturu鈥檚 physics textbook from years ago.听
Beyond physics
The team also designed its product with feedback from real students studying physics and with science teachers. One educator was impressed by the tool鈥檚 ability to help students lead their 鈥渙wn little experiments.鈥�
The tool isn鈥檛 perfect yet, Suzuki noted. So far, it can only turn a brand new diagram into an effective simulation about 60% of the time. But the team is working to improve those success rates.
Ultimately, the researcher and his colleagues would like to move beyond physics鈥攗sing similar technology to transform any diagram in any kind of document into a vivid, 3D world that students can step into with the aid of augmented reality glasses.
鈥淭hat鈥檚 our goal鈥攖o escape from static 2D into the interactive 3D world,鈥� Suzuki said.
Gunturu鈥檚 mom, meanwhile, can鈥檛 wait for him to design a tool that works on biology textbooks.
鈥淭hat would be her dream come true,鈥� he said.
Other co-authors of the new study included Rubaiat Habib Kazi at Adobe Research; Nandi Zhang and Jarin Thundathil at the University of Calgary; and Yi Wen at Texas A&M University.
