8.13.1. Workflow of SfM

Multiple overlapping photos are captured from various angles—often using UAVs or handheld cameras.

In the first step of the Structure from Motion (SfM) workflow, we gather numerous photographs that overlap with each other. These images can be taken with Unmanned Aerial Vehicles (UAVs), commonly known as drones, or with handheld cameras. The importance of overlapping photos is that they provide different perspectives of the same scene, which is crucial for understanding the spatial arrangement of objects. When different viewpoints are captured, it allows us to reconstruct a three-dimensional representation of what is observed.

Imagine you are trying to create a 3D model of a sculpture. Instead of taking just one picture, you walk around the sculpture and take multiple photos from different angles. Each image captures a different side of the sculpture, helping you understand its full shape. This is similar to how SfM works with overlapping photos.

Algorithms such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features) detect unique points (keypoints) in images and match them across multiple views.

Next, we use sophisticated algorithms to identify and match points of interest in the overlapping images. These points are referred to as 'keypoints'. Algorithms like SIFT and SURF can recognize these keypoints regardless of changes in scale or rotation. Once identified, matching keypoints across different images helps to establish the common features and links between those images. This is essential for determining how the images relate to one another spatially, which facilitates the 3D reconstruction process.

Think about how you recognize a friend in a crowd from different angles. Even if they turn around or are standing at an angle, you still see their unique features: their hairstyle, clothing, and height. The algorithms in SfM work similarly by identifying and matching these unique features across images so that the 3D structure can be reconstructed accurately.

Intrinsic (focal length, sensor size) and extrinsic (position and orientation) parameters of the camera are estimated using bundle adjustment.

After matching keypoints, it is important to estimate the camera’s position and orientation when each photograph was taken. This process uses mathematical techniques called 'bundle adjustment', which refines the camera parameters to minimize errors in the 3D point cloud generated from the matched points. Intrinsic parameters refer to characteristics specific to the camera itself, such as focal length, whereas extrinsic parameters relate to the camera's position in space and its direction at the time of capturing the image. Understanding these parameters is crucial for accurately aligning the reconstruction.

Imagine trying to find your way around a city using a map. You must understand both your current location (extrinsic) and the details of the map itself (intrinsic). The better you estimate your position and how you are looking at the map, the more accurately you can navigate the city. Camera pose estimation operates similarly by figuring out the camera's position and orientation relative to the scene.

3D coordinates of matched features are triangulated to create a sparse model.

Once the camera poses have been estimated and keypoints matched, the next step is to create a sparse point cloud. This involves triangulating the positions of the matched features in 3D space, giving each feature a specific set of coordinates. This sparse model contains a limited number of points spread throughout the scene, which represents the basic structure of the observable data.

Think of this process like constructing a frame for a house. The sparse point cloud is similar to the skeleton or the beams that hold the house together, providing a basic structure upon which more details can be added. Just like you would add walls and a roof later, the sparse model serves as the foundation for further 3D reconstruction.

Multi-view stereo (MVS) algorithms convert sparse clouds into dense 3D point clouds.

The sparse point cloud is just the beginning of the 3D model. To create a more detailed and intricate model, we employ Multi-view stereo (MVS) algorithms. These algorithms analyze the overlapping images to add more points to the existing sparse cloud, resulting in a dense point cloud that captures finer details and features within the scene. Dense reconstruction essentially fills in the gaps, providing a richer representation of the 3D structure.

Imagine a jigsaw puzzle. At first, you might see just a few pieces laid out (the sparse point cloud), but as you start adding more pieces and filling in the gaps, the complete picture becomes clearer (the dense point cloud). Similarly, MVS helps to create a comprehensive 3D model from the initial sparse data.

The point cloud is converted into a mesh and textured using original images.

Finally, to create a visually realistic and usable 3D model, the dense point cloud is transformed into a mesh structure. This mesh acts like a skin that covers the underlying points and represents the object's surface. After the mesh is created, textures are applied using the original images taken during the photo acquisition phase. This gives the 3D model a lifelike appearance.

Consider this as creating a sculpture: first, you build the basic shape with clay (the mesh), and then you paint it and add details to make it visually appealing (the texture). Thus, the finished model is not only structured well but also visually accurate.

--