基于圖像的大規(guī)模三維重建課件

1、Towards Large Scale 3D Reconstruction from ImagesOverviewIntroductionLarge scale 3D reconstruction pipelineLarge scale Structure-from-MotionLarge scale Multiple View StereoConclusion3D ReconstructionReconstruct 3D models of scenes and objects by multiple images automaticallyInputOutputWhy 3D Reconst

2、ruction?Store: complete space informationBrowse: complete and real feeling of scenes with more freedom for users Reproduce: automatically modeling tool for 3D modeler and source for 3D printerApplication:Digital museums and citiesVirtual realityMovie industry Mapping and navigationAltizureA custom c

3、loud platform for 3D reconstructionUser can upload their captured images to obtain 3D reconstruction results automaticallyProvide solutions for large scale 3D reconstructionLarge Scale 3D ReconstructionMergeLarge Scale 3D Reconstruction PipelineLarge scale imagesFeature DetectionImage RetrievalDBFea

4、ture MatchingRelative PosesCamera RegistrationCamera Poses and Sparse PointsGlobal Bundle AdjustmentIMUGPSGCPMulti-sensor FusionCameraSelectionand ClusteringDenseReconstructionDense PointsDenseReconstructionDenseReconstructionMergeSurfaceSurface Recon-structionStructure-from-Motion(SfM)Multiple View

5、 Stereo(MVS)Structure-from-MotionLarge scale imagesFeature DetectionFeature DetectionTraditional handcraft featureSIFT, Harris Conner, etcLearning based featureLIFTLack of generalizationNot for 3D reconstruction taskGeoDescNetwork: L2-NetLearning from SfM results and coarse mesh triangulated from sp

6、arse pointsTraining sample from patches survived from SfM and non-survived but with high similaritiesPublication: Z. Luo, T. Shen, L. Zhou, S. Zhu, R. Zhang (Corresponding author), Y. Yao, T. Fang, and L. Quan, “GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints,” in ECCV 2018Ge

7、oDescPublication: Z. Luo, T. Shen, L. Zhou, S. Zhu, R. Zhang (Corresponding author), Y. Yao, T. Fang, and L. Quan, “GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints,” in ECCV 2018GeoDescStructure-from-MotionLarge scale imagesFeature DetectionImage Retrieval.DatabaseImage Retri

8、eval for SfMFeature based vocabulary tree to retrieve relative imagesLearning based retrievalGoal: overlapping region instead of semantic similarityTraining data: hundreds of reconstructed models in AltizureIntroduce true overlapping information to lossPublication: T. Shen, L. Zixin, L. Zhou, R. Zha

9、ng, S. Zhu, T. Fang, and L. Quan, “Matchable Image Retrieval by Learning from Surface Reconstruction,” in ACCV 2018Structure-from-MotionLarge scale imagesFeature DetectionImage Retrieval.DatabaseFeature MatchingGraph-based MatchingGoalReduce candidate match pairs before geometry validationRobust to

10、ambiguityPublication: T. Shen, S. Zhu, T. Fang, R. Zhang, and L. Quan, “Graph-based consistent matching for structure-from-motion,” in ECCV 2016Graph-based MatchingMethodCamera graph initialized by a minimum spanning tree based retrieval rankingsGraph Expansion by Strong TripletsCommunity-Based Grap

11、h ReinforcementPublication: T. Shen, S. Zhu, T. Fang, R. Zhang, and L. Quan, “Graph-based consistent matching for structure-from-motion,” in ECCV 2016Structure-from-MotionLarge scale imagesFeature DetectionImage Retrieval.DatabaseFeature MatchingRelative PosesCamera RegistrationCamera RegistrationRe

12、gister all cameras into a global systemIncremental methodsSelect next view by the number of points (Snavely et al., 2006)Skeletal set (Snavely et al., 2008b): minimum camera set representing the whole sceneDistributed matching & registration (Agarwal et al., 2011)AdvantageEasy to filter outliersDisa

13、dvantageLocal information resulting in driftSlowCamera RegistrationLarge Scale Distributed Camera RegistrationCombine incremental and global registrationDivide camera graphIncremental registration in subgraphsMotion averaging to mergePublication: S. Zhu, T. Shen, L. Zhou, R. Zhang, T. Fang, and L. Q

14、uan, “Accurate, scalable and parallel structure from motion,” CoRR, vol. abs/1702.08601, 2017.Large Scale Distributed Camera RegistrationCamera clustering based on GraphSize constraintCamera number in each cluster is not too largeCamera numbers are balanced among clustersCompleteness constraintOverl

15、apping ratio is enoughAlgorithmDividing by Normalized-cutExpansion for completeness constraintLarge Scale Distributed Camera RegistrationLarge Scale DistributedGlobal Motion AveragingDivide camera graph while remaining all relative posesNormalized-CutsEdge weight: common feature corresponding number

16、Publication: R. Zhang*, S. Zhu*, L. Zhou, T. Shen, T. Fang, P. Tan, and L. Quan, “Very large-scale global sfm by distributed motion averaging,” in CVPR 2018. (*: equal contribution)Large Scale DistributedGlobal Motion AveragingLocal optimize inner-variables in subgraphsGlobal optimize inter-variable

17、s with similarity transformations Publication: R. Zhang*, S. Zhu*, L. Zhou, T. Shen, T. Fang, P. Tan, and L. Quan, “Very large-scale global sfm by distributed motion averaging,” in CVPR 2018. (*: equal contribution)Large Scale DistributedGlobal Motion AveragingCamera Registration ResultsStructure-fr

18、om-MotionPublication: R. Zhang, et al, “Distributed very large scale bundle adjustment by global camera consensus,” in ICCV 2017 (Oral) & PAMI 2018 .Large scale imagesFeature DetectionImage Retrieval.DatabaseFeature MatchingRelative PosesCamera RegistrationCamera Poses and Sparse PointsGlobal Bundle

19、 AdjustmentIMUGPSGCPMulti-sensor FusionGlobal Bundle AdjustmentIn-Core MethodLinearization and LM algorithmFactorization and preconditioning : S. Agarwal, etc. Bundle Adjustment in the Large, ECCV, 2010 Parallel bundle adjustment: C. Wu, etc. Multicore Bundle Adjustment, CVPR, 2011Bundle Adjustment

20、V.S. Motion AveragingBundle AdjustmentMotion AveragingConvexityNon-convexConvex for some objective functionsInitializationGood initializationRandom initializationScaleAll cameras and pointsOnly Cameras or with sampled pointsTopological structureBipartite graphGeneral graphCamerasPointsOut of Core Me

21、thodOut of core bundle adjustment (Kai Ni, ICCV 2007)Divide and Conquer, iteratively optimize non-overlapping and overlapping blocksTransfer all cameras and points of overlapping blocksHigh I/O overhead and densely connected camera graphs are hard to be splitOut of Core MethodDistributed bundle adju

22、stment (A. Eriksson, CVPR 2016)Proximal Splitting MethodTransfer all points of overlapping blocksI/O of points is still too high for very large scale bundle adjustment and cameras cannot share intrinsic parametersADMM for Distributed OptimizationHidden variablesHelper variablesAugmented Lagrangian M

23、ultipliersDistributed Bundle AdjustmentConsensus on cameras instead of points (splitting points instead of cameras):Lower I/O overheadFaster convergencePublication: R. Zhang, et al, “Distributed very large scale bundle adjustment by global camera consensus,” in ICCV 2017 (Oral) & PAMI 2018 .Hidden v

24、ariablesDistributed Bundle Adjustment5Augmented Lagrangian MultipliersPenalty parametersHelper variablesDistributed Bundle Adjustment6ConvergenceImplementationExtensions to improve convergence ratesSelf-adaption penaltyOver-relaxationBlock splitting in practiceCut camera visibility graph by Normaliz

25、ed-Cut Reduce the number of overlapping camerasBalance the size of each blockFor very large scale data-set, first use KDTree to split blocks so that Normalized-Cut can workDataset scaleDataset# cameras# points# projections# blocksBuildings510260k1.40M32Final 961961187k1.69M32Roman Forum1084158k1.12M

26、32Street1130347k2.00M64Ladybug1723157k679k64Venice1778994k5.00M64Piccadilly2152136k9.20M64Trafalgar5288214k1.82M128Final46M29.0M256Town3642827.8M3512M1024City138193100.2M10088M2048I/O Overhead1 A. Eriksson, J. Bastian, T.-J. Chin, and M. Isaksson. A consensus-based framework for distrib

27、uted bundle adjustment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016I/O Overhead1 A. Eriksson, J. Bastian, T.-J. Chin, and M. Isaksson. A consensus-based framework for distributed bundle adjustment. In The IEEE Conference on Computer Vision and Pattern Recognit

28、ion (CVPR), June 20161Convergence curveOur methodOur method without relaxationOur method without self-adaptionMethod of 11 A. Eriksson, J. Bastian, T.-J. Chin, and M. Isaksson. A consensus-based framework for distributed bundle adjustment. In The IEEE Conference on Computer Vision and Pattern Recogn

29、ition (CVPR), June 2016Large Scale SfM ResultsCityCity BTownStructure-from-MotionLarge scale imagesFeature DetectionImage Retrieval.DatabaseFeature MatchingRelative PosesCamera RegistrationCamera Poses and Sparse PointsGlobal Bundle AdjustmentIMUGPSGCPMulti-sensor FusionFusion with GCPGround control

30、 pointsMeasured by accurate GPS on siteManual matchingTwo kinds: for adjustment and for evaluationFor adjustmentUsing visual result to triangulate GCP matchingAlignment visual result to GPS systemAttending global bundle adjustment with high weightEvaluation by GCPFor evaluationEvaluated by the dista

31、nce between the position from triangulation and on-site measurementGCP evaluation resultsDataTown1621201.650.19City3052541.390.15Structure-from-MotionPublication: R. Zhang, T. Fang, S. Zhu, and L. Quan, “Multi-scale tetrahedral fusion of a similarity reconstruction and noisy positional measurements,

32、” in ACCV 2014Large scale imagesFeature DetectionImage Retrieval.DatabaseFeature MatchingRelative PosesCamera RegistrationCamera Poses and Sparse PointsGlobal Bundle AdjustmentIMUGPSGCPMulti-sensor FusionPositional Measurement FusionPositional Measurement FusionPositional measurements (PM)Camera pos

33、es with PMCamera poses without PM(c) Problematic initial alignment(a) Drift-free noisy positional measurementsx0 xkx1x2(b) SfMc0c1c2cjTetrahedral Fusion Publication: R. Zhang, T. Fang, S. Zhu, and L. Quan, “Multi-scale tetrahedral fusion of a similarity reconstruction and noisy positional measuremen

34、ts,” in ACCV 2014c0c1c3c2Relative Pose TermMulti-scale SamplingTetrahedral constraintsRelative pose constraintsCamera posesPosition measurements(a) Tetrahedral constrains at scale s0(b) Tetrahedral constrains at scale s1(c) Relative pose constrains at scale s0(d) Relative pose constrains at scale s1

35、Simulation ExperimentDataMethodWALLTetra12.335.60319.5312.535.47520.17IBA103.056.08176.2103.056.05176.2FLIGHTTetra3.3861.8339.8851.4810.7583.051IBA3.2991.9009.7711.6081.0113.261(i)(ii)(iii)(iv)(v)(i)(ii)(iii)(iv)(v)WALLFLIGHTReal Video Experiment(ii)(iii)(a) NEW(i)(i)(ii)(iii)(b) GARDEN(i)(ii)(iii)(

36、c) CAMPUS(i)(ii)(iii)(d) KITTIStructure-from-MotionPublication: L. Zhou, S. Zhu, Z. Luo, T. Shen, R. Zhang (corresponding author), T. Fang and L. Quan, “Learning and Matching Multi-View Descriptors for Registration of Point Clouds,” in ECCV 2018.Large scale imagesFeature DetectionImage Retrieval.Dat

37、abaseFeature MatchingRelative PosesCamera RegistrationCamera Poses and Sparse PointsGlobal Bundle AdjustmentIMUGPSGCPMulti-sensor FusionMergeMulti-SfM MergeMulti-view local descriptorLearned by Fuseption-ResNetRobust MatchingBy belief propagationVertex: point correspondenceEdge: compatible or incomp

38、atible K-nearest neighborPublication: L. Zhou, S. Zhu, Z. Luo, T. Shen, R. Zhang (corresponding author), T. Fang and L. Quan, “Learning and Matching Multi-View Descriptors for Registration of Point Clouds,” in ECCV 2018.Large Scale 3D Reconstruction PipelineLarge scale imagesFeature DetectionImage R

39、etrievalDBFeature MatchingRelative PosesCamera RegistrationCamera Poses and Sparse PointsGlobal Bundle AdjustmentIMUGPSGCPMulti-sensor FusionCameraSelectionand ClusteringDenseReconstructionDense PointsDenseReconstructionDenseReconstructionMergeSurfaceSurface Recon-structionStructure-from-Motion(SfM)

40、Multiple View Stereo(MVS)Multiple View StereoCamera Poses and Sparse PointsImageSelectionand ClusteringDenseReconstructionDense PointsMergeSurfaceSurface ReconstructionDenseReconstructionDenseReconstructionImage Selection & ClusteringImage selection: Multi-view Compression Image clustering: CMVSSele

41、ct images before clusteringCluster images based on image featuresMerge and filterExisting problemsRedundantNon uniformMixed qualityPublication: S. Zhu, T. Fang, R. Zhang, and L. Quan, “Multi-view geometry compression,” in ACCV 2014Jointly Space Division and Image SelectionDivide space and select ima

42、ges for each regionSpace is represented by coarse meshes from sparse points(a) Points and cameras from SfM(b) Coarse mesh(c) Mesh segments and camera clusters(d) Dense points of each cluster(e) Final Dense points Publication: R. Zhang, S. Li, T. Fang, S. Zhu, and L. Quan, “Joint camera clustering an

43、d surface segmentation for large-scale multi-view stereo,” in ICCV 2015Criteria for Space Division and Image SelectionHierarchical Division & OptimizationImage selection based on EM for K clustersMaximize coverage under size constraintSpace division by MRF to K segmentsSimultaneously consider covera

44、ge and smoothnessHierarchical Division & OptimizationImage selection based on EMImage selection based on EMSpace Division by MRFtriangular meshlabels of trianglesImplementationInitialization the segmentation by K-MeansVery large scale data setsCompromise between optimum and efficiencyFirst split the

45、 space so that each region can be viewed by about 2000 cameras by K-MeansPrepare for dense reconstructionFor each cluster, reproject its segment to each image as masksDilate masks for robustnessQuantitative ResultsDatasetTempleDinoBasilicaColosseumMethodOursCMVSOursCMVSOursCMVSOursCMVS37779962322452

46、1824362.263.671.401.582.313.110.7241.00P1.0801.0851.031.041.091.421.011.121.301.281.391.3297.594.972.168.5DatasetTreviCampusDubrovnikExplanationMethodOursCMVSOursCMVSOursCMVS123472719148526123270Compactness4.575.462.403.221.024.17UniformityP1.001.601.111.312.2010.4Redundancy49.234.01519.71416.4137.7

47、898.23CompletenessEfficiencyQualitative ResultsOursCMVS(a)(b)(c)Qualitative ResultsSegmentationOursCMVSOursCMVSBasilicaColosseumTreviSegmentationOursCMVSOursCMVSDubrovnikCampusTemple & DinoQualitative ResultsMultiple View StereoCamera Poses and Sparse PointsImageSelectionand ClusteringDenseReconstru

48、ctionDense PointsMergeSurfaceSurface ReconstructionDenseReconstructionDenseReconstructionDense ReconstructionMVSNet (by Yao Yao, ECCV 2018)A deep learning based multiple view dense reconstruction methodIntroduce the idea of plane sweeping into CNN/pdf/1804.02505.pdf Surface ReconstructionSurface ref

49、inement Shiwei Li, et al. Efficient multi-view surface refinement with adaptive resolution control. ECCV 2016Thin structure surface reconstructionShiwei Li, et al. Reconstructing Thin Structures of Manifold Surfaces by Integrating Spatial Curves. CVPR 2018. Image Selection for Surface Reconstruction

50、Using results by the proposed method directlySegment the fine meshes according to the results - not easy to implementUsing images selected to each segmentUsing space division result by KDTreeSelect images for each segment by the proposed EM based methodSegment is in good order - easy to implementCon

51、clusionLarge scale 3D reconstruction challengesAmbiguityGraph-based consistent matchingScalabilityDistributed camera registrationDistributed global motion averaging Distributed bundle adjustmentImage clustering for MVSRedundancy and mixed qualityImage selection for MVSConclusionLearning New SfM resu

52、lts from oldNew descriptors from oldNew retrieval model from oldTake traditional geometry method in network 3D reconstruction from coarse to fineLarge scale may induce more problemsFuture worksOther problems induced by large scale image dataLight and environment changesReal time 3D reconstructionEnd

53、-to-end deep learning based 3D reconstructionWe want you!Welcome to apply for Prof. Quans PhD students and join in Altizure!Welcome to join in Tencent Youtu X-Lab as internship or regular employee!Thank you!Questions?Slide show of sample outputPublicationsZ. Luo, T. Shen, L. Zhou, S. Zhu, R. Zhang*, T. Fang and L. Quan, “Learning Local Descriptors by Integrating