Simple Technique for Slicing and Segmentation of 3D Avatars

What is a 3D avatar?

With the evolution of new technologies, human lifes are getting smarter and easier. Similarly nowadays 3D body scanning technology is evolving and many 3D scanners are now available under reasonable prices with good quality scans. Even now the Apple iPhone has a LiDar scanner which is very convenient and easy to use.

Using a 3D scanning application on your smartphone or even by proper 3D scanning device you can scan a digital representation of the object or person. The scan has a point cloud with depth and the color etc. This point cloud can be converted to mesh which can be opened in a 3D modeling application.

These scanned 3D models can be consumed by various applications. One of them  is 3D Measure Up which uses a 3D avatar of a human body data to identify landmarks and extract measurements.

What is this blog about?

In this blog, we discuss a simple technique to slice and segment 3D avatar. I will describe the technology components that are used to perform the slicing and segmentation. 

As a first step, we use 3D Measure Up to detect and extract human body measurements e.g. height, waist size, leg length etc. Once we have those, we can easily slice the 3D avatars and segment the body parts by using any open source library which supports mesh boolean operations like PyMesh.

What technology components are used? 

We have used the following technology components to achieve our objective:

1. 3D Measure Up: is a service that makes it easy to add powerful 3D measurement and analysis to your applications. It lets you easily build powerful applications to identify and measure many landmarks on a 3D scan of a human body.

3D Measure Up is an anthropometric measurement service that detects various landmarks or features of a human body such as ankle, knee, tip of nose, shoulder point etc. It also allows you to measure various parameters of these landmarks e.g. position of mid-thigh from the ground, girth of biceps, waist, bust, volume of torso etc. 3D Measure Up is based on the proven, highly scalable, machine learning technology developed by ProtoTech’s 3D geometry, graphics and visualization experts.

2. PyMesh: is a rapid prototyping platform focused on geometry processing. It provides a set of common mesh processing functionalities and interfaces with a number of state-of-the-art open source packages to combine their power seamlessly under a single developing environment.

Mesh process should be simple in python. PyMesh promotes human readable, minimalistic interface and works with native python data structures such as numpy.ndarray.

How to implement?

There are two separate tasks that you have to perform to accomplish this task. First is implemented using 3D Measure Up and the second task is implemented using Pymesh.

This task can be done in a single application or can be done in two separate microservices.

Below are the steps to utilize 3D Measure Up and PyMesh to slice a 3D model of the human body and extract body parts:

1. 3D Measure Up to get all the measurement:

• Create an application using 3D Measure Up.
• Provide a 3D human body part file (mesh format – example STL file) as an input to 3D Measure Up.
• 3D Measure Up will clean up mesh, do all the measurement and output you with all the landmarks and measurement information (like – ankle, knee, tip of nose, shoulder point, position of mid-thigh from the ground, girth of biceps, waist, bust, volume of torso etc)
• By using the 3D Measure Up output you can create a cubical or circular bounding for that body part that you want to slice and provide it as input to the PyMesh task.

2. PyMesh to slice the human body part:

• Create an application which imports PyMesh.
• Provide a 3D human body part file (example STL file) as an input to PyMesh and open the file to read the mesh.
• Now create another mesh in PyMesh using the bounding box information we calculated using 3D Measure Up.
• And then perform a mesh boolean intersection operation between both meshes.
• You will get sliced human body parts as a result.

Anything that I should watch out for?

1.  Building and installing PyMesh on windows, we have used it on Linux OS as it has good documentation for Linux but you can also use it docker container.
2.  Creation of a presized bounding box from the data you get from 3D Measure up is an important task.
3.  Handling different poses, postures and body shapes could be challenging.
4.  Slicing the arm from the shoulder could be difficult for the fat shaped human body.

Results: