Design considerations for Fitting Room “Magic Mirror” Technology
By Antão Almada
A mirror where people can see themselves wearing virtual clothing has always been considered magical or something from sci-fi movies. Given the quick advances in technology, expectations are very high. People expect it to be fully established by now.
There are already many examples of “virtual try-on” (VTOs) based on augmented reality. For example, the Browns’ mobile app lets users try a pair of sneakers or a watch just by pointing the smartphone camera at their feet or wrist.
These VTOs let users try an accessory on a specific body part, but the accessories don’t deform during the experience. It was still a big challenge to carry out this project. It required high curation and finding the right partners.
Full-body VTOs featuring delicate items are still very scarce. The reason is that these are much harder to implement for many reasons. This article highlights why that is.
Nowadays, it’s also possible to scan a person with a smartphone, change their clothing, and rotate and zoom using touch gestures. That’s a different type of experience that doesn’t involve significant technical challenges. This article will focus on the interactive “magical” mirror experience.
Applications
It’s very tempting to deploy these VTOs just to showcase the technology, for the hype. It may offer a novelty effect in the short term, but users may quickly give up on using it, and the investment would be in vain.
One thing these VTOs won’t be able to reproduce in the foreseeable future is the sense of touch or haptics. Trying on clothing is not just a visual experience. The customer wants to feel the materials and how it fits. So, what value can the VTO add to compensate for the lack of the rest of the experience?
First, we must consider where this type of VTO will be experienced.
A smartphone or a tablet are not the ideal devices to experience a full-body VTO. Their form factor works better for trying accessories on the head, neck, wrist, hands, or feet. But it’s almost impossible to hold the smartphone camera and see the whole body unless it’s pointing to a mirror or using a selfie stick.
Full-body VTOs are best experienced on something resembling a wall’s large vertical mirror. Something that you would find in a bedroom, a walk-in closet, or a fitting room in a store. Currently, this type of device would most likely be found in a store.
Next, we consider what problems the VTO helps solve.
Most often VTOs enable customers to try items not available in the store. The question has been: do luxury fashion physical stores want an “infinite shelf”?
The hygiene constraints resulting from the current pandemic may generate more demand for VTOs, as customers may not be allowed to try as many physical items as before. VTOs will enable the customer to try multiple virtual combinations and select a smaller set of material things to try in the fitting room
Mirror Reflections
For a mirror to be “magical,” it has to perform two basic tasks. The reflection of the person must behave as in any regular mirror, and it must render an augmented set of virtual clothes.
These effects can be achieved by using a two-way glass mirror with a display behind it (an “optical mirror”). The mirror will reflect the customer’s image, while the display features the virtual garment. The physical properties of the two-way mirror blend these two images.
Please note that the customer’s reflection will still include their actual clothing. They won’t disappear magically. To increase the contrast of the virtual items and fade the real clothing, we require a high-luminance display and dimmed ambient lights.
This type of mirror results in a single-user experience. When the customer is accompanied by other people, they won’t see the virtual items aligned with the body reflection. Also, the customer won’t be able to focus their eyes simultaneously on their reflection and the virtual clothing. They will perceive their reflection to be twice the distance from their virtual clothing.
A fully digital solution may provide a better experience. They can offer a large display without the optical illusions of a two-way glass mirror.
There are two possible digital solutions. The display shows the image captured by a camera pointed at the customer (“augmented reality mirror”), or it shows an avatar that looks and moves just like the customer (“avatar mirror”).
In both cases, for the customer to understand it intuitively as a mirror, their “reflection” must behave exactly as in a real mirror. For example, when the customer raises their right hand, the “reflected” right hand must raise accordingly. When the customer is near the mirror, their “reflection” must be more significant than when far away.
For the best results, with the “augmented-reality mirror,” the video camera should have the same resolution ratio as the display and the highest resolution possible. Full-body VTOs should be mounted in portrait mode (tilted 90 degrees) at the average height of the eyes of the customers.
It should also be as close to the center of the display as possible; a camera above the mirror would distort the image of the customer with a large head and tiny feet. The ambient space should also be well-lit to avoid shadows and grain in the camera image.
Body-Pose Estimation
The virtual garment must be correctly rendered, given the customer’s body pose at any given moment and the movement’s flow. The position and orientation of the torso, limbs, and head must be estimated to the best accuracy possible with the best refresh rate possible.
We prefer to avoid attaching sensors to the customer’s body if possible, as they interfere with the flow of their experience. It should work just like any regular mirror: the customer steps in front of the display and looks at it.
To achieve this, a camera must be used with a full-body skeletal tracking algorithm to track the position of all body joints. In the case of the “augmented reality mirror,” it can be the same camera used to capture the image.
These algorithms require the limbs to be visible on the camera image to work well. They won’t estimate correctly when they are only partially visible or if the customer wears loose sleeves, loose pants, or long skirts. Also, the customer may have physical disabilities. When the pose estimation fails, the mirror should not show the “reflection” in awkward body positions.
The pose estimation should work with whatever background and more than one person in the field of view. The application should be robust enough to “dress” only one person (maybe starting with the one in the center) and allow them to move around without the clothing jumping to another person, even when they are swapping positions.
Using a camera may raise privacy concerns. For this and other reasons, FARFETCH avoids placing cameras in fitting rooms. A full-body VTO would only be used on the shop floor, where the customer does not take off their clothes.
Virtual Garments
Images of virtual garments must be generated and adapted to the customer’s position at any given moment. These images should be as accurate as possible, but we may have to compromise for several possible reasons. I’ll try to address most of them.
2D Images
The simplest solution is to use pictures of the items and render them in front of the corresponding body part in the customer’s image. A digital version of the old “paper doll dressing books.” This looks awkward when the customer moves and doesn’t allow them to turn around. Still, this solution is used on some existing VTOs. It’s relatively inexpensive, the content creation is very simple, and it requires very modest hardware.
3D Rigid Bodies
The use of 3D rigid bodies allows the user to turn around. This solution is adequate for rigid items like shoes, bags, jewelry, and other accessories. It can be used for clothing, but it looks awkward when the customer moves, just like the solution with the 2D images.
Creating 3D models is a big challenge. FARFETCH has an extensive catalog and adds hundreds of new items each day. The process must be cost-effective and scalable. The time it takes to create each item model must be as short as possible without compromising quality. Shooting and processing 3D imagery will always take more time than it does for 2D pictures.
One of the processes used to get a 3D rigid model is called “photogrammetry.” It generates a 3D model from multiple images of the object from different points of view. It can also capture the colors and patterns, something not all 3D scanners can do. To accelerate this process, all the pictures can be taken simultaneously using a rig that positions several cameras around the physical item.
3D scanning has several challenges:
- Contactless scanning processes work great if the object is convex but may have issues when parts of the item are occluded by other elements. This may result in holes in the 3D model.
- Scanning precision can be affected by light-reflective or light-absorbing materials.
- Texture scanning “bakes” the reflections to the environment surrounding the items and for each camera position. These will not look realistic once in the simulation environment, where items and viewers can move relative to the light sources.
- Photogrammetry may not work for repetitive patterns.
These issues must be fixed in post-processing, adding costs to the process.
Still, this process has excellent results for some items like sneakers and handbags. We’re sure this type of scanning will keep progressing, and the quality will improve for a wide range of fashion items.
Rendering 3D models require a more powerful computer with a Graphical Processing Unit (GPU). That means that the mirror enclosure may need to be larger, or the computer must be hidden elsewhere.
3D Soft Bodies
For a realistic simulation of clothing, the garment should not be rigid. It must conform to the body shape, pose, gravity pull, stitching, material type, and thickness. The garment’s model must have multiple components stitched together, just like real clothing.
The models generated by 3D scanning are usually not suitable for this type of simulation. They have very irregular meshes and are almost impossible to manipulate.
Manually created models are usually better suited for this. There are many 3D modeling applications, and currently, some are developed explicitly for clothing design. These follow a similar pipeline to the physical garment design, with much quicker design iterations and no material waste.
Still, the manual creation of a digital model takes a long time, making it very expensive. It’s not feasible to reproduce at scale existing physical garments.
Several brands already use this type of software to design their own line of clothing. It would cut a lot in the cost of the VTOs if the brands shared or licensed the models they use for their production.
Soft Clothing Physics
Simulating the behavior of soft clothing requires a simulation engine. Physics engines that simulate fluids and cloth require a lot of processing power. Only recently have these become real-time, which is necessary for an interactive experience. Nowadays, physics engines take advantage of the power of the graphical processing unit (GPU). This means it requires a powerful GPU that can handle both the rendering and the physics.
When planning to have this on the shop floor, this powerful computer’s dimensions have to be considered.
Avatars
An avatar is a digital representation of the customer. The avatar would only be visible for the “avatar mirror,” but it’s still an essential part of the “optical mirror” and “augmented reality mirror.” The avatar will give shape to clothing when using cloth physics. It should have the same body shape, proportions, height, and weight as the customer.
Skeletal tracking provides distance between the body joints, allowing an approximation of body height and proportions. The camera could be used to obtain other measurements, such as body shape, but the customer must wear very tight clothing. This process raises privacy issues, and many stores prefer not to subject the customers to this. Less intrusive alternatives will have to be used, but getting them right is challenging.
Occlusion
Occlusion in 3D rendering means that, like in the real world, everything behind something else will not be visible. The parts of the virtual items behind the customer’s body should not be rendered. These can be things like:
- the watch strap behind the wrist
- the back of the garment behind the neck
- the inside of the sleeves
- the hands and arms in front of a garment
This is handled automatically when everything in the scene is virtual. That’s the case for the “avatar mirror” but not the other “mirrors.” There are two possible solutions to handle the occlusion of those different types:
- Create a 3D object based on the depth information from a depth-sensing camera.
- Use the avatar.
In both cases, a “shader” has to be applied so that the 3D occlusion object does not appear in the color buffer (invisible) but does appear in the depth buffer (occludes).
For this to be imperceptible, the occlusion must be wholly aligned with the customer’s image. The depth of information must be exact, or the avatar must be very similar to the customer’s body shape.
Unfortunately, high-resolution depth-sensing cameras mean a lower frame rate, adversely affecting the experience’s interactivity.
Proof-Of-Concept (POC)
Shortly before the Covid lockdown, we started working on a POC to explore the current feasibility of a full-body VTO. We developed it on Unity, using Deform Dynamics for the cloth physics, MakeHuman for the avatar, CLO for the garment, Nuitrack for the skeletal tracking, a depth-sensing RealSense D435i camera, and a GeForce RTX 2070 SUPER graphics card. Putting all these components together, it was relatively easy to achieve a satisfactory result. Here is a video we recorded during lockdown where a better-suited model was not widely available: https://youtu.be/XoGn0ac82Kc
We found the frame rate and interactivity to be very decent. The skeletal tracking is good when it works. Sometimes it would fail and rip apart the garment, which is unacceptable.
We were pretty satisfied with the result but well aware that it would need a lot more work to achieve the quality required to put it in a store. Still, there were two major issues we did not consider in this POC: how to get the dimensions of the customer body and how to produce digital garments at scale.
Generative Adversarial Network (GAN)
In recent years, research around GANs has evolved amazingly. This is the basis for “deep fakes”: synthetic media where a person in an existing image or video is replaced with someone else’s likeness.
Research has also been done on applying this technology to VTOs. Just like with image and video “deep fakes,” the result of the VTO is based on models “learned” from previous cases and adapts a new set of data.
This could potentially solve the problems with the traditional computer graphics approach described above. The garment could be generated from a minimal picture set, fit the body shape accurately and move without requiring a physics engine. It could estimate the body shape from a picture of the customer even without having to wear tight clothing.
It could also remove the “reflection” from the real clothing the customer is wearing so that it won’t be seen, for example, on the neckline.
Interactivity
It is also essential to consider how the customers will interact with the mirror. They won’t be at arms-reach of the mirror. A touch interface would require the customer to walk back and forth whenever they want to interact. Touchless alternatives should be considered, such as gesture detection and voice recognition.
Conclusion
We have no doubt that the full-body VTO will be a handy tool in the fashion industry’s future. It won’t replace the traditional fitting room but will complement it in several scenarios.
Technology has evolved immensely in recent years, but there are still some issues that must be solved. Mainly the production of digital garments at scale.
Many technical compromises are still required. It’s impossible to precisely align the garment and customer reflection with the “optical mirror.” It is challenging to get accurate occlusion with the “augmented reality mirror.” The “avatar mirror” solves those issues, but the customer does not see their reflection.
Just as the light bulb is not the incremental evolution of the candle, the use of GANs is a very different approach to the technical challenges of a full-body VTO. They may solve the issues with the traditional computer graphics solution.
The full-body VTO is something we may not have in the stores in the immediate future, but we are steadily evolving in that direction.
We are working on many more disruptive innovations. The FARFETCH platform is continuously evolving to power the future of retail.
Originally published at https://www.farfetchtechblog.com on February 21, 2023.
