We have smartphones, tablets, and laptops. They're all equipped with an unprecedented number of cameras and even 3D sensors. And yet, if you want to just take a picture of your cat and put her onto a thank-you card, that process is incredibly convoluted. You might take the photo on your phone. Email it to your computer. Then import it into Photoshop.

But a new smartphone and desktop app coming from Cyril Diagne, an artist-in-residence at Google, greatly simplifies the process. Called AR Copy Paste, it can literally take a photo of a plant, mug, person, newspaper—whatever—on your phone, then, through a magical bit of UX, drop that object right onto your canvas in Photoshop on your main computer.

You kind of have to see it to get it. So take a look:

4/10 – Cut & paste your surroundings to Photoshop

Code: https://t.co/cVddH3u3ik

Book: @HOLOmagazine
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓#ML #AR #AI #AIUX #Adobe #Photoshop pic.twitter.com/LkTBe0t0rF

— Cyril Diagne (@cyrildiagne) May 3, 2020

"Recently, during confinement, I've spent some time every weekend to prototype new ideas on this topic of interaction design + machine learning," says Diagne. "They are small prototypes that I post on my Twitter, along with technical insights and link to the source code for anyone to try."

AR Copy Paste began in just this way. Diagne mocked up the project during off hours, then posted it to Twitter . . . where it was promptly liked over 40,000 times. He also shared the technology that made it possible. AR Copy Paste uses an object recognition algorithm called BASNet to capture an object and cut it out from its background. Then Diagne uses another piece of code, called SIFT, to handle the paste.

[Image: courtesy Cyril Diagne]

How does that paste work? It's actually the result of two photos taken at once, Diagne explains. When you hit the paste button in the app, your phone's camera takes a photo of your monitor, and your computer takes a screen grab of what's showing on the monitor. So for this effect to work, you need software installed on both your mobile device and your computer. The software cross-references these two images to figure out your position in relation to the computer screen and create that perfect paste moment.

After seeing the viral popularity of the tweet, Diagne began working to create a standalone smartphone and computer app that will allow anyone to use AR Copy Paste, no coding knowledge required. The smartphone app prototype will be free to test when it comes out. You can sign up to be among the first to try it here.

"It's not so hard to open the scope of its potential applications. First of all, it's not limited to objects, and it works equally well with printed materials like books, old photo albums, and brochures," says Diagne. "Likewise, the principle is not limited to Photoshop but can be applied to any image, document, or video-editing software."

Diagne even teases that AR Copy Paste could be used for functions other than merely copying and pasting. He imagines that you might use its core software to point to objects around you in a certain sequence to perform a function. I imagine you could build a password tool in such a way, treating objects as a code. Rather than "1, 2, 3, 4″ your passcode could be a series of things you aim your phone's camera at—"succulent, feet, mouse, lamp."

Perhaps Diagne's app will take off, and perhaps it won't. But at the very minimum, the project is a reminder: The user interface of our world hasn't been perfected yet. We have the technology to improve it. Now, we just need to keep up the imagination.