Have you ever needed to insert an image of something in real life into a file online? Duh, what is this, computers for beginners? Well, no longer will that process require taking a cell phone photo, texting or emailing it to myself, converting the file type, and copy-and-pasting or inserting the image. I’m already exhausted by how so very 2019 all that fuss sounds.
A new augmented reality demo going viral on Twitter and GitHub demonstrates an application that can cut-and-paste real life images into a computer program, taking a ton of manual effort out of the process of image editing.
The demo shows a person taking a photo on an Android phone of a household object, like a notebook or a plant, which then creates an isolated image of that object with the background removed. The camera then points to photoshop on a Mac laptop and “pastes” the image into the program. Though there are a few seconds of delay between each action, the result is remarkable.
The secret sauce here is BASNet (Qin et al, CVPR 2019) for salient object detection and background removal.
The accuracy and range of this model are stunning and there are many nice use cases so I packaged it as a micro-service / docker image: https://t.co/jbAmO89cRy pic.twitter.com/DGe9IffPtS
— Cyril Diagne (@cyrildiagne) May 3, 2020
Advertisement. Scroll to continue reading.
The app was created by programmer-in-residence at the Google Arts & Culture Lab in Paris Cyril Diagne. The 34-year-old artist and designer created the app for fun and didn’t expect the enthusiastic reaction from the Internet that it got. “It’s part of a series of experiments I’m doing every weekend to explore how machine learning and AI can help create more digital interactions that are more natural. Basically using the algorithm’s ‘intelligence’ to remove the layers of abstractions we had to build in digital interfaces up to now,” Diagne told Cult of Mac.
Diagne explained on Twitter that BASNet is the critical technology that reads the edges of an object and separates it from its surroundings. OpenCV SIFT tracks where the cell phone camera is pointing to find the object for to cut.
Of course, the application is still a prototype and at the moment only works with Photoshop. It also requires creating a local server to link the app to Photoshop.
This is not the first application to cut-and-paste real world images either; Google Lens started offering similar technology to cut-and-paste images of text in 2018. Nonetheless, using AR and computer learning to create and move images in a matter of seconds is a simple and elegant solution to a common technological challenge. Interested users can get started with the demo using the instructions on GitHub.
Heather Buffo is a Cleveland native, a recovering Bostonian, and an Austin newbie. Heather is the Venture Growth & Partnerships Lead at Republic where she works with partners in private investing to democratize access to capital for entrepreneurs. Heather studied neurobiology at Harvard University, and is a City Year Boston AmeriCorps alum. She likes to write for AG, drink Austin beer, and ride around town on her road bicycle. His name is Pippin. Say hello if you see them.
