In the 40s of the 20th century, Maria spent her childhood in Barceló, Spain. Her memories of her father are still vivid.
When she was six years old, when Maria wanted to see her father, she would go to a neighbor's house in the same building. Because she could see the prison below through the balcony railing there, and catch a glimpse of her father through the small window of the cell. Her father was imprisoned there for opposing the government at the time.
There are no photographs of this memory from the balcony, but now Maria can get something similar: a "fake photo". For the design studio Domestic Data Streamers (DDS) in Barcelona, this is a "memory-based reconstruction", that is, the reconstruction of the scene in the memory that might be captured by a real photograph.
Fake photos are blurry and distorted, but they can still instantly travel back to that moment in memory.
"It's easy to tell if the restoration of a memory is accurate because it's a visceral response. Pau Garcia, founder of DDS Studios, said, "This happens every time. The reaction was, 'Ah, yes, that's it!'"
（来源：DOMESTIC DATA STREAMERS）
Currently, DDS Studios operates the "Synthetic Memory" project, which has transformed the memories of dozens of people into pictures in this way. The studio uses image generation models, such as OpenAI's DALL-E, to visualize people's memories.
Since 2022, the studio, which has received funding from the United Nations and Google, has been working with immigrant and refugee communities around the world to transform scenes that have never been photographed into images, or to recreate long-lost old photos for those who have been uprooted from their homes.
Now, DDS Studio is taking over a building next to the Barcelona Design Museum, documenting people's memories of the city with synthetic images. Garcia said anyone can participate and contribute memories to the growing archive.
Synthetic memory may be more than just a social or cultural endeavor. In the summer of 2024, the studio will begin working with researchers to see if its technology could be used to treat dementia.
Unforgettable graffiti
The idea for the project came from a 2014 experience Garcia had when he was working in Greece with an organization that was relocating Syrian refugee families.
One woman told him that she was not afraid to become a refugee, but she was afraid that her children and grandchildren would become refugees because they might forget their family history: where they shopped, what they wore and how they wore them.
Garcia asked volunteers to graffiti the walls of the building where the families lived, drawing memories of the woman. "Those graffiti were terrible, but the idea of synthetic memories was born. He said.
A few years later, when Garcia saw what the image generation model could do, he remembered the graffiti: "It was the first thing that came to my mind. ”
（来源：DOMESTIC DATA STREAMERS）
The process employed by Garcia and his team is simple. They sat down with the interviewee and asked the latter to recall a particular scene or event. A prompt engineer writes prompts for the model based on recalls, resulting in an image.
His team has built a vocabulary of prompt terms that have been validated and adapted to evoke memories from different historical periods and different locations.
But Garcia says the prompts often have to be tweaked back and forth: "You show the resulting image to the interviewee, and they might say, 'Oh, the chair is on the other side' or 'It's at night, not during the day.'" Then you have to keep revising the prompt until it matches the interviewee's memory. ”
To date, DDS Studios has used this technology to preserve the memories of various immigrant communities, including Korean, Bolivian and Argentine families living in São Paulo, Brazil.
At the same time, it has also partnered with a nursing home in Barcelona to study how this memory reconstruction technique can help the elderly.
The team, in collaboration with researchers in Barcelona, conducted a small trial of 12 subjects to apply this approach to recall therapy. It is a treatment for dementia that aims to stimulate cognitive abilities by showing someone images of the past.
Recall therapy was developed in the 60s of the 20th century and has many supporters, but researchers have mixed opinions on its effectiveness and methods of implementation.
Garcia said this pilot project allowed the team to refine the process and ensure that participants were able to sign the informed consent form.
The investigators plan to conduct a larger clinical study in the summer of 2024 with colleagues at the University of Toronto, Canada, comparing therapies employing image-generating models with other treatments.
One of the things they found in the pilot was that older people would have a better connection with the images if they were printed out.
"When they see a picture on a screen, they can't make the same emotional connection," Garcia said. But when they can see it with their own eyes, memory becomes even more important. ”
Blur is better
The researchers also found that the old version of the image generation model performed better than the new version. They used two models launched in 2022 at the beginning of the project: the DALL-E 2 and the Stable Diffusion. The latter is a free image generation model released by Stability AI.
The images produced by these models are flawed, including bizarre faces and distorted bodies. But when they switched to the latest version of Midjourney, another model that can create more detailed images, the results weren't all that popular.
"If you make something that's very realistic, people pay attention to the details that aren't there," Garcia said. Conversely, if it is vague, the concept of memory reconstruction is better understood.
Memories are a bit like dreams, they are not photographs, there are no nuanced details. You may not remember whether the chair was red or green, only that there was a chair. ”
Figure | When they are able to see it with their own eyes, memory becomes even more important. (Source: @MARCASENSIO_FOTO)
Since then, the team has been working with the old model again. "For us, blemishes are a characteristic," Garcia said.
Sometimes, things may or may not exist. This is the quantum state in the picture, which fits very well with the characteristics of memory. ”
Sam Lawton, an independent filmmaker who was not involved in the work, is excited about the project. He is particularly pleased that the team will be studying the cognitive impact of these images in a rigorous clinical study.
Lawton used image generation models to recreate his own memories. In one of his 2023 films called Expanded Childhood, he uses DALL-E to expand old family photos beyond the scene, blurring real childhood scenes with surreal scenes.
"The effect that exposure to this generated image has on the human brain is what motivated me to make this film," Lawton says. I couldn't do a full research work, so I turned to storytelling that was most natural to me. ”
Lawton's work addresses many questions: What are the effects of long-term exposure to AI-generated or altered images?
Do such images help reconstruct traumatic memories, or do they create a false sense of reality that leads to confusion and cognitive dissonance?
Lawton showed his father photos from Extended Childhood and added his comment to the film: "I feel something is wrong, but I don't know what it is. Could it be that I don't remember?"
（来源：DOMESTIC DATA STREAMERS）
Garcia knows the dangers of confusing subjective memories with real photographic records. He said the memory-based reconstruction work done by his team was not a "document of fact."
In fact, he points out that this is another reason to stick with older image generation models to produce less realistic images.
"It's important that people can tell very clearly what is a synthetic memory and what is a real photo," Garcia said. It's a simple way to tell the difference. ”
But Garcia now worries that the company behind the models may take older versions off the shelves. Most users expect bigger and better models. For synthetic memories, the less (detail), the better.
"I'm really scared that OpenAI is going to shut down DALL-E 2 and we're going to have to use DALL-E 3," he said. ”
Support: Ren
Operation/Typesetting: He Chenlong