To provide better supervision during training and encourage generalization, we develop a tool and methodology that enables the creation of a scalable semi-synthetic RGB-D dynamic scene dataset, which we call REFRESH. The network that learns the per-pixel rigidity also solves for the relative camera pose between two images, and we can accurately refine the pose as a least square problem with the learned dense flow correspondences and rigidity region.

, which we evaluate together with the rigidity network to estimate both ego-motion and scene-motions. To solve for 2D correspondences, our framework relies on 2D optical flow, and is not tied to any particular algorithm. 2, takes a sequential image pair with color and depth (RGBD) as the input and mainly focuses on dynamic scenes with a moving camera (e.g., panning), where camera motion and objects motions are entangled in each observation. A brief example of our results is shown in Fig. Unlike other methods solving the problem with piecewise rigid motion , clustering local motions , and semantic segmentation , our network can infer per-pixel rigidity by jointly learning rigidity and the relative camera transform from large-scale dynamic scene data.

This is mainly because the disambiguation of camera motion (ego-motion) from object motion requires the correct identification of rigid static structure of a scene. 3D motion field estimation in a dynamic environment is, however, a challenging and still open problem when the scene is observed from different view points and the amount of coverage of moving objects in each image is significant. This task is commonly referred as 3D motion field or scene flow estimation. The estimation of 3D motion from images is a fundamental computer vision problem, and key to many applications such as robot manipulation , dynamic scene reconstruction , autonomous driving , action recognition , and video analysis . Through our evaluation, we show the proposed framework outperforms current state-of-the-art scene flow estimation methods in challenging dynamic scenes. For training and testing the rigidity network, we also provide a new semi-synthetic dynamic scene dataset (synthetic foreground objects with a real background) and an evaluation split that accounts for the percentage of observed non-rigid pixels. With the learned network, we show how we can effectively estimate camera motion and projected scene flow using computed 2D optical flow and the inferred rigidity mask. Compared to other state-of-the-art 3D scene flow estimation methods, in this paper, we propose to learn the rigidity of a scene in a supervised manner from an extensive collection of dynamic scene data, and directly infer a rigidity mask from two sequential images with depths. The primary challenge is the disambiguation of the camera motion from scene motion, which becomes more difficult as the amount of rigidity observed decreases, even with successful estimation of 2D image correspondences. In real-world applications, a dynamic scene is commonly captured by a moving camera (i.e., panning, tilting or hand-held), increasing the task complexity because the scene is observed from different viewpoints. Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems.

Rigid motion