Designers and architects are faced with complex challenges when creating objects such as furniture, cars, and buildings in our 3D world. For example, chairs must be comfortable to sit on, while cars need to be aerodynamic. To ensure that shapes are functional in real-world settings, designers use computer-aided design software to generate 3D models and simulation software to test their designs, such as a wind simulation on a 3D model of a car to ensure it flows well. 

While 3D modeling provides precise and realistic depictions of objects and surroundings, designers often delay using it until their ideas are fully developed. This is in part because creating 3D shapes using 2D input devices such as a computer mouse on a flat surface is difficult. 

“As humans, we have the ability to recognize shapes even if they are drawn on flat piece of paper”, says Adrien Bousseau. “For instance, when we see three lines meeting at a corner, we can easily perceive it as a cube. However, for a computer, these three lines are just flat lines. Determining where to lift the pen strokes to create a 3D model is an arduous task. Tricks that artists use for optical illusion are impossible for a computer to understand.”

Sketches

Adrien Bousseau and his team set out to tackle this problem by aiming to enhance the early stages of design exploration. Their goal was to develop a method for generating 3D models from sketches, allowing designers to convey their concepts to computers in the same manner as they would to clients or colleagues. This would enable them to assess, for instance, the aerodynamic properties of a car's bodywork immediately following its sketching.

A first step was to spend some time among professional designers. “Designers draw to explain their shapes to others, and they have developed techniques to draw these shapes as clearly as possible”, says Bousseau. “Therefore, for us, it was key to understand these techniques. We studied drawing books and interviewed designers while observing how they draw. Additionally, we collaborated with two design teachers who helped us understand these techniques.”

Geometric constraints

Bousseau and his team looked at 400 drawings to examine the geometric properties of various types of lines, and used the techniques developed by professional designers in order to represent 3D shapes. “A first stream of research we did was to try to identify many of those geometric constraints, from lines in the drawings, and then run an algorithm that searches for the shape and respects those constraints as best as possible. “

For instance, when people draw a cube, they draw lines that converge at a point, as this is how people draw in perspective. Lines that converge towards the same vanishing point should be parallel in 3D. While computer vision algorithms rely on some of these properties to reconstruct 3D shapes from photographs, line drawings pose unique challenges because they are imperfect. Therefore, we developed specific algorithms that combine several geometric cues extracted from the drawings to reconstruct 3D shapes.”

Bousseau and his team also delved into the realm of machine learning by teaching an algorithm to identify shapes. “This involved presenting the algorithm with numerous drawings and their corresponding 3D shapes. To ensure the algorithm learned how to predict the 3D shape accurately when given a drawing, we needed many pairs of drawings and shapes.”

However, amassing thousands of such pairs proved challenging as hiring a professional designer to draw so many cubes, cylinders, and complex shapes would be time-consuming. “As an alternative, we used computer graphics techniques to generate synthetic drawings, similar to those used in creating computer-animated movies.”

Creative freedom

Bousseau and his team have developed the first methods that enable the automatic reconstruction of freehand design drawings made on a pen tablet. Prior to their research, existing techniques for transforming drawings into 3D models were inadequate because they required significant effort from the designer, according to Bousseau. 

“You needed to trace the drawing with a specific tool in a particular software. The algorithm required guidance from the designer to identify parallel lines, making the process laborious. As a result, these tools were not widely adopted. We wanted to surpass that state of the art by allowing designers to have the greatest possible freedom in their drawing process. Rather than developing a method and testing it with users, we began by gathering drawings without any clear idea of how to reconstruct them in 3D. The drawings represented diverse drawing styles, and from there we set our goal to preserve that creative freedom for the user.”

“This proved to be quite challenging, as we discovered that people draw in many different ways and sometimes with inaccuracies. However, we also noticed that designers used same types of techniques. Thanks to this observation, we were able to create an algorithm that accurately reconstructed a majority of the collected drawings, which was a great achievement.”

Improve drawings

The technologies has multiple potentials, particular in design education. Beginner designers often struggle with perceiving objects in 3D. With the ability to switch effortlessly between drawing and 3D, they can overcome this challenge. Moreover, designers may not always draw accurately, which will show in the 3D reconstruction, says Bousseau. “If a cube is drawn slightly wider than it should be, it will appear more elongated when rotated in 3D. The algorithm could potentially correct these mistakes, and help the designer to improve their drawings.”

Bousseau and his group are working on integrating their tool into Blender as Open Source software, and are currently reaching out to industrial partners, although any plans for technology transfer will depend on the feedback received. Meanwhile he is planning to take his research even a step further. 

“As we continued our observations of people drawing, we noticed a similarity between their approach and the way professional 3D modelers construct shapes. For instance, instead of beginning with the final silhouette of a car, they would use basic shapes like cubes and cylinders to build up to the details. This realization led us to the idea of not only recreating the drawing in 3D but also translating the drawing steps directly into 3D modeling software commands.“

“Drawing is a visual language, and drawing techniques almost resemble a vocabulary. Recent breakthroughs in machine learning have led to significant success in language translation. We are now exploring whether similar tools can be used to translate between visual languages, specifically from drawings to 3D shapes. We have already seen impressive progress in natural language interfaces, such as ChatGPT. Allowing people to express themselves in their natural language and having the computer understand it has opened up many new possibilities. We believe that the same will be true for drawings.“

Biography

Adrien Bousseau is a researcher at Inria Université Côte d'Azur, in the GraphDeco research group (inria.fr). He did his Ph.D. at Inria Rhône-Alpes and his postdoc at UC Berkeley. He also did several internships at Adobe Research. Bousseau received one of the three Eurographics 2011 Ph.D. award for his research on expressive image manipulations, and a young researcher award from the French National Research Agency for his work on computer-assisted drawing. In his ERC Starting Grant project, he worked with a talented team:  Yulia Gryaditskaya, Felix Hähnlein and Tibor Stanko (geometric constraints for shape reconstruction); Changjian Li and Bastien Wailly (machine learning for shape recognition); Emilie Yu (3D drawing in virtual reality), and David Jourdan (simulation and manufacturing for rapid prototyping).