It turns out that the rabbit wearing the AI-generated top hat is just the tip of the iceberg.
Google is the latest phone company to launch an AI-powered photo-editing tool this year, following Samsung’s slightly unsettling but mostly delightful Sketch-to-Image feature and Apple’s seemingly much tamer Image Playground this fall. The answer for the Pixel 9 is a new tool called “Reimagine,” and after a week of using it with a few of my colleagues, I’m more convinced than ever that none of us are prepared for what’s coming.
Reimagine is a logical extension of last year’s Magic Editor tool, which lets you select and delete parts of a scene, or change the sky to a sunset-like effect. There’s nothing shocking about this. But Reimagine doesn’t just go one step further, it breaks the whole thing. You can select any non-human object or part of a scene and enter a text prompt to produce something in that space. The result is often Very Believable, even incredible. Lighting, shadows, and perspective generally match the original photo. Of course, you can add fun things like wildflowers or rainbows or something like that. But that’s not the problem.
A few of my colleagues helped me test the boundaries of Reimagine with Pixel 9 and 9 Pro review units, and we found that it produced some very disturbing stuff. Some of them require some creative hints to get around the obvious guardrails; if you choose your wording carefully, you can make it create a pretty convincing body beneath the blood-stained sheets.
During our week of testing, we added car wrecks, smoking bombs in public places, sheets that appeared to cover bloody bodies, and drug paraphernalia to the images. This looks bad. Mind you, this isn’t some special software that we use on purpose – it’s all built into the phone my dad can walk into and buy at Verizon.
When we asked Google to comment on the issue, company spokesperson Alex Moriconi issued the following statement:
Pixel Studio and Magic Editor are useful tools designed to unleash your creativity with text-to-image generation and advanced photo editing on your Pixel 9 device. We design generative AI tools to respect the intent of user prompts, which means they may create potentially objectionable content when directed by users. In other words, nothing progressed. We have clear policies and terms of service about the types of content we allow and don’t allow, and we have guardrails in place to prevent abuse. At times, tips may challenge the guardrails of these tools, and we remain committed to continually enhancing and improving the safeguards we have in place.
To be sure, our creative prompts to bypass filters clearly violate these policies. It’s also against Safeway policy to label organic peaches as conventionally grown at the self-checkout, but I don’t know anyone who would do that. People with the worst intentions won’t care about Google’s terms and conditions. The most troubling thing about all of this is the lack of powerful tools to identify this kind of content on the web. Our ability to create problematic images far exceeds our ability to identify them.
When you edit an image with Reimagine, there’s no watermark or any other obvious way to indicate that the image was AI-generated – just a tag in the meta. That’s all well and good, but standard metadata can be easily stripped from an image by simply taking a screenshot. Moriconi told us that Google uses SynthID, a more powerful tagging system, for images created in Pixel Studio because they are 100% synthetic. But images edited using Magic Editor will not get these tags.
To be sure, doctored photos are nothing new. People have been adding strange and deceptive things to images since the beginning of photography. But the difference now is that it’s never been easier to actually add these things to your photos. A decade or two ago, adding a convincing car crash to an image required time, expertise, knowledge of Photoshop layers and expensive software. Those barriers are gone; now all it takes is a few words, some time, and a new Pixel phone.
It’s also easier than ever to quickly spread misleading photos. The tools to manipulate your photos convincingly exist within the same device you use to capture them and publish them for the world to see. We uploaded a “reimagined” image to Instagram Stories as a test (and quickly deleted it). Meta didn’t automatically mark this as AI generated, I’m sure if no one saw it they would be wiser.
Who knows, maybe everyone will read and abide by Google’s AI policies and use Reimagine to add wildflowers and rainbows to their photos. That’s great! but just in case they Noit’s best to be a little skeptical of the photos you see online.
