TextSLAM: Visual SLAM with Planar Text Features

Boying Li, Danping Zou∗, Daniele Sartori, Ling Pei, and Wenxian Yu

Abstract— We propose to integrate text objects in man-madescenes tightly into the visual SLAM pipeline. The key idea of ournovel text-based visual SLAM is to treat each detected text as aplanar feature which is rich of textures and semantic meanings.The text feature is compactly represented by three parametersand integrated into visual SLAM by adopting the illumination-invariant photometric error. We also describe important detailsinvolved in implementing a full pipeline of text-based visualSLAM. To our best knowledge, this is the first visual SLAMmethod tightly coupled with the text features. We tested ourmethod in both indoor and outdoor environments. The resultsshow that with text features, the visual SLAM system becomesmore robust and produces much more accurate 3D text mapsthat could be useful for navigation and scene understanding inrobotic or augmented reality applications.

I. INTRODUCTION

Visual SLAM is an important technique of ego-motionestimation and scene perception, which has been widely usedin navigation for drones [1], ground vehicles or self-drivingcars [2], and augmented reality applications [3]. A typicalvisual SLAM algorithm extracts point features [4], [5] fromimages for pose estimation and mapping. Recent methods[6] [7] even directly operate on pixels. However, it is wellknown that incorporating high-level features like lines [8],or even surfaces [9] in the visual SLAM system will lead tobetter performance with fewer parameters.

One type of object surrounding us that can be used ashigh-level features is text. Text labels in daily scenes offerrich information for navigation. They can help us recognizethe landmarks, navigate in complex environments, and guideus to the destination. Text extraction and recognition havebeen developing fast in these days [10] [11] because of thebooming of the deep neural networks and the emergenceof huge text datasets such as COCO-Text[12], DOST[13],and ICDAR[14]. One question raises whether texts can beintegrated into a visual SLAM system to not only yield betterperformance but also generate high-quality 3D text maps thatcould be useful for navigation and scene understanding.

There are several attempts towards text-aided navigationand location in recent years. A navigation system [15], [16]for blind people, assisted with text entities, is built uponthe visual-inertial SLAM system shipped on the GoogleTango tablet. Similarly, Wang et al. proposed a method toextract text features [17], which are then used for fusionwith Tango’s SLAM outputs to facilitate closing loops. Theaforementioned works have shown the great potential ofintegrating text features with existing SLAM systems, though

All the authors are with Shanghai Key Laboratory of Navigation andLocation-based Service, Shanghai Jiao Tong University. ∗Correspondingauthor: Danping Zou (dpzou@sjtu.edu.cn)

Fig. 1. TextSLAM in a shopping mall. Left: Detected texts in the images(in yellow rectangles) and the zoomed-in view of 3D text map. Right: 3Dtext maps and camera trajectory in top-down view. The text objects areillustrated in pink boxes and their normal directions are shown in blue.

in a loosely coupled manner. It is worth further investigationinto a tightly coupled approach by putting texts into theSLAM pipeline instead of treating the existing SLAM systemas a black box.

We propose a novel visual SLAM method tightly coupledwith text features. Our basic motivation is that texts areusually planar and texture-rich patch features: Texts wespotted in our daily life are mostly planar-like regions, atleast for a single word or character if not the whole sentence.The rich pattern of a text entity makes the text object anaturally good feature for tracking and localization. It isdemonstrated in our work that through fully exploring thosecharacteristics of text features, we can improve the overallperformance of the SLAM system, including the quality ofboth localization and semantic map generation. The maintechnical contributions of this paper includes:

1) A novel three-variable parameterization for text featuresis proposed. The parameterization is compact and allowsinstantaneous initialization of text features with small motionparallaxes.

2) The text feature is integrated in the visual SLAM systemby adopting the photometric error measured by normalizedsum of squared differences. Such photometric error is robustto illumination changes and blurry images caused by quickcamera motions. Tracking and mapping of text features aredone by minimizing the photometric errors without extra dataassociation processes.

3) We present details for implementing such a text-based SLAM system, including initialization and update oftext features, text-based camera pose tracking and back-endoptimization. To our best knowledge, this is the first visualSLAM method with text features tightly integrated.

We conduct experiments in both indoor and outdoor en-vironments. The results show our novel text-based SLAM

arX

iv:1

912.

0500

2v1

[cs

.CV

] 2

6 N

ov 2

019

method achieves better accuracy and robustness, and pro-duces much better 3D text maps than does the baselineapproach. Such semantically meaningful maps will benefitnavigation in man made environments.

II. RELATED WORK

a) Planar features: Planar features have been studiedin visual SLAM community since the early stage. In the earlyworks [3][18][19], the planes in the scene were detected byRANSAC [20] among estimated 3D points and employed asnovel features to replace points in the state. It requires muchfewer parameters to represent the world using planes insteadof points, hence significantly reduces the computationalcost of estimation. Those works show that planar featuresimprove both accuracy and robustness of a visual SLAMsystem. However, existing methods require 3D information todiscover the planar features, usually using a RGB-D camera[21][22]. This becomes difficult using only image sensors.An interesting idea [23] is to assume each image patchsurrounded the detected feature point as one observation of alocally planar surface. The assumption however seldom holdsin realistic scenes, as feature points might be extracted fromanywhere in the scene. Nevertheless, texts in realistic scenesare mostly located on planar surfaces, though sometime onlylocally. Therefore texts are naturally good planar features thatcan be easily detected by text detectors.

b) Object features: Integrating object-level featuresinto visual SLAM systems has been receiving increasinginterest in recent years [24][25][26]. Existing methods how-ever require pre-scanned 3D models to precisely fit theobservation on the image. Though recent work [27] at-tempted to reconstruct the 3D model of objects online witha depth camera, it is still difficult to be generalized tounseen objects with only image sensors. Another solutionis adopting 3D bounding boxes [28][29] or quadrics [30]to approximate generic objects. This however suffers fromloss of accuracy. Unlike generic objects, the geometry oftext objects is simple. As aforementioned, text objects canbe treated as locally planar features and also contain a richamount of semantic information about the environment. Itis hence worth investigating how to design a SLAM systembased on the text objects.

c) Text-aided navigation: Text is a naturally good vi-sual fiducial marker or optical positioning to assistant navi-gation [31][32] yet the technique is still under development[31]. Several loosely coupled text-aided navigation methodshave shown a great potential for perception, navigation, andhuman-computer interaction. The early work [33] used theroom number as a guidance for robot to autonomously movein the office-like scenes. The authors [34] annotated the textlabels on the existing map generated by a laser SLAM systemto help robots understand each named location. In the work[17], the spatial-level feature named ’junction’ is extractedfrom text objects, and then combined with the location andmapping output of Tango’s SLAM system at the stage of loopclosing. The authors present a text spotting method [15] forthe assistant navigation system relying on the SLAM system

Fig. 2. A text object is compactly parameterized by θ. The inverse depthof a text point p can be computed by ρ = θTm and its projection onto thetarget view Ct is a homography transform with respect to the relative poseTth between the two views.

shipped on Google Tango Tablet. Similarly, with the SLAMsystem provided via Google Tango, a mobile solution [16] ofassistant navigation system combines various sources, suchas text recognition results, human-machine interface, speech-audio interaction, for blind and visually impaired people totravel indoor independently.

Existing text-aided methods regard SLAM systems (eithervision-based or laser-based) as a black box and takes lessattention on the accuracy of 3D text maps. By contrast, theproposed method integrates the text objects tightly into theSLAM system, and focus on generating high accurate textmap that can be used for future pose estimation and loopdetection.

III. TEXT FEATURES

A. Parameterization

As discussed previously, most text objects can be regardedas planar and bounded patches. Each text patch (usuallyenclosed by a bounding box) is anchored to the cameraframe, named as host frame, when it is firstly detected onthe image as shown in Fig. 2. Expressed in the coordinatesystem of the host frame, the plane where the text patchlies is described by the equation nTp + d = 0, wheren = (n1, n2, n3)T ∈ R3 is the normal of the plane andd ∈ R is related to the distance from plane to the origin ofthe host frame; p ∈ R3 represents the 3D point on the plane.

A straightforward parameterization of a text plane couldbe directly using the four parameters (n1, n2, n3, d) of theplane equation. But this is a over parameterization that leadsto rank deficient in nonlinear least-squares optimization. Wepropose to use a compact parameterization that contains onlythree parameters.

θ = (θ1, θ2, θ3)T = n/d. (1)

We’ll show that this parameterization is closely related to theinverse depth of the 3D point on the text plane.

Within the host frame, each 3D point p ∈ R3 observed onthe image is able to be represented by its normalized imagecoordinates m = (u, v)T and its inverse depth ρ = 1/h ,where the depth h is the distance between this 3D point andthe camera center. The 3D coordinates of this 3D point arecomputed as p = (uh vh h)T = hm, where m denotes thehomogeneous coordinates of m. If the 3D point locates on

the text plane, we have h ·nTm+ d = 0. The inverse depthρ of this 3D point is then computed as

ρ = 1/h = −nT/d m = θTm. (2)

That is, we can use a simple dot product to quickly infer theinverse depth of a text point from its 2D coordinates, giventhe text parameters θ.

On the other hand, if we have at least three points onthe text patch (for example, three corners of the boundingbox), with their inverse depths, we can quickly obtain thetext parameters by solvingm

T1

...mTn

θ =

ρ1

...ρn

, n ≥ 3. (3)

This allows us to quickly initialize the text parameters fromthe depth value of three corners of the text bounding box.

To fully describe a text object, properties such as theboundary of the text object, pixel values, and the semanticmeaning are also kept in our system. Those information canbe acquired from a text detector as we’ll described later.

B. Projection of the 3D text object onto a target image

Here we describe how to project a 3D text object anchoredto a host frame onto the target image related to the targetframe. Let Th,Tt ∈ SE(3) represent the transformationsfrom the host frame and the target frame to the world framerespectively. The transformation from the host frame to thetarget frame is T = T−1

t Th. We let R, t be the rotationand translation of T. Given the text parameters θ and theobserved image point m (with the homogeneous coordinatesm) in the host frame, the 3D coordinates of point p are :

p = m/ρ = m/(θTm). (4)

The point is then transformed into the target frame. Let thetransformed point be p′. We have p′ ∼ Rm + t mTθ.Here notation ∼ means equivalence up to a scale. Usingthe pinhole camera model, the coordinates of image pointm′ = (u′, v′)T of p′ in the target frame are computed as

u′ = (rT1 m+ t1m

Tθ))/(rT3 m+ t3m

Tθ))v′ = (rT

2 m+ t2mTθ))/(rT

3 m+ t3mTθ))

, (5)

where r1, r2, r3 are the row vectors of R and t =[t1, t2, t3]T. Note that (5) is in fact the homography trans-formation from m to m′, where the homograpny matrix isdefined as H ∼ R + tθT. Hence, the whole process ofprojecting a 3D text object on the image plane of a targetframe can be described as a homography mapping

m′ = h(m,Th,Tt,θ). (6)

C. Photometric error for text objects

Photometric error is used to compare the projected textobject and the observed one on the image. This is done bypixel-wise comparison and similar to the direct approaches[7] which have been shown to be accurate and robust withoutfinding corresponding feature points explicitly. The biggestissue of using photometric error is to handle the intensity

changes. Existing work [7] adopts an affine model to addressintensity changes, but it requires extra parameters involvedin optimization and sophisticated photometric calibration toget good performance.

We choose to use zero mean normalized cross-correlation(ZNCC) as the matching cost to handle illumination changes.Let Ω be the set of pixels within the text region. Let m ∈ Ωbe a text pixel. The normalized intensities for text pixelsare: I(m) = (I(m) − IΩ)/σΩ, where IΩ and σΩ indicatethe average intensity and the standard deviation of the pixelswithin the text region. The text patch in the host frame andthe predicted one in the target frame (6) are then comparedby : ZNCC(Ih, It) =

∑m∈Ω Ih(m)It(m

′). The ZNCCcost is between −1 and 1. The larger ZNCC cost indicatesthe two patches are more similar. However, it is difficult todirectly use the ZNCC cost in visual SLAM, since it cannot be formulated as a nonlinear least squares problem. Wetherefore adopt a variant form of ZNCC as the cost function

E(Ih, It) =∑m∈Ω

(Ih(m)− It(m′))2. (7)

Though the cost function is similar to the SSD (Sum ofSquared Difference) cost, it contains additional normalizationprocess making it robust to illumination changes. If weexpand this cost function as :∑

m∈Ω

(Ih(m)2 + It(m′)2)− 2

∑m∈Ω

Ih(m)It(m′), (8)

we’ll discover that minimizing this cost function is equivalentto maximizing the ZNCC cost, since in the first term,∑Ih(m)2 and

∑It(m

′)2 are two constants related to thepixel number in Ω. The photometric error of a text object πwith respect to the target frame k is defined as :

Eπ,tphoto =∑

m∈Ωπ

φ((Ih(m)− It(h(m,Th,Tt,θπ)))2), (9)

where φ(·) is the Huber loss function to handle possibleoutliers. Here, we use Ωπ to represent the text region on theimage plane in the host frame. As we’ll describe later, tomake the computation faster, we do not use all the pixelswithin the text region, instead select only some of them asreference pixels to compute the photometric error.

IV. TEXTSLAM SYSTEM

Our TextSLAM system is built upon the basic system us-ing point features and adopts the keyframe-based frameworkto integrate the text features tightly. Fig. 3 illustrates theflowchart of TextSLAM. we’ll detail the key components inthe following sections.

A. Initialization of text objects

Text objects are extracted on the image whenever a newimage has been acquired. We use the EAST[35] text detectorto extract text objects. Some examples of text extraction aredemonstrated in Fig. 1. The outputs are arbitrary-orientationquadrilaterals enclosing the text regions. Once a text objecthas been extracted, we detect some FAST [36] feature pointswithin in the text region and track them until the next key

Fig. 3. An overview of the TextSLAM system.

frame by Kanade-Lucas-Tomasi(KLT) [37] tracker. Then theparameter of text object is initialized from the tracked points.Let mi ↔ m′i be the corresponding points in both views,and R, t be the transformation between the two frames. From(5), we have

[m′i]×tmTi θ = −[m′i]×Rmi (10)

where mi and m′i are the homogeneous coordinates of mi

and m′i. Note that the rank of the matrix on the left handside is one. It requires at least three pair of correspondingpoints to solve θ. The text parameter is futher refined byminimizing the photometric error as defined in (9).

After initialization, we keep the text quadrilateral withthe text object. The four corners can be projected ontoother views (6) to predict the appearance of this text object.Note that a text object can only be initialized when itsquadrilateral in the current view is not intersected withexisting text objects or partially out of the image. Otherwisethey are rejected as ’bad initialization’.The newly initializedtext objects are kept being updated in the following frames.They are inserted into the map only if the text object hasbeen observed in at least nmin (5 in our implementation)frames.

B. Camera pose estimation with text objects

Both points and text objects in the map are involved incamera pose estimation. The camera pose estimation is tominimize the following cost function

E(Tt) = Epoint(Tt) + λwEtext(Tt), (11)

where Tt ∈ SE(3) represents the current camera pose.Epoint and Etext come from the feature points and the textfeatures respectively. Note that Epoint consists of geometricerrors, or reprojection errors, but Etext contains only photo-metric errors

Etext =∑π,t

Eπ,tphoto. (12)

The trade-off between them needs to be regulated by theweight λw since they are in different units (position differ-ence vs intensity difference).

The weight λw is computed as λw = σrep/σphoto. σreprepresents the standard deviation of the reprojection error of apair of corresponding points (in both x and y directions) andσphoto represents the standard deviation of the photometricerror of a text object as defined in (7). Those standarddeviations can be acquired through a small set of trainingdata (given corresponding points and text patches).

Optimization of the cost function (11) is a nonlinear leastsquares problem. As the photometric cost Etext is highlynonlinear, it requires a good initial guess of Tt to avoidbeing trapped in a local minimum. We firstly use a constantvelocity model to predict the camera pose. Based on thisprediction, we then apply a coarse-to-fine method which willbe described later to optimize the text cost Etext alone to getthe initial guess of Tt. Finally, using this initial guess, wecan estimate the camera pose by minimizing (11) iteratively.

C. Bundle Adjustment with text objects

We apply bundle adjustment from time to time in a localwindow of key frames similar to [4]. The cost function ofbundle adjustment consists the point part and the text part :

E(x) = Epoint(x) + λwEtext(x). (13)

The cost function resembles that of camera pose estima-tion while involves more parameters to be optimized. Thevariable x include the camera poses of key frames in thelocal window, the 3D coordinates of point features, and thetext parameters. We also adopt a coarse-to-fine method tooptimize (13) as in camera pose estimation.

Though we may use all the pixels within the text region toevaluate the photometric errors, a better way is to use only asmall part of them to allow for fast computation. We selectrepresentative pixels with the maximum number of 15 byusing the FAST detector [36], and use these reference pixelsto evaluate photometric errors.

V. EXPERIMENT

A. Data collection

We collected a set of image sequences for evaluation inboth indoor and outdoor scenes. Fig. 4 shows the devicewhich we used for data collection. It consists of the hand-held camera and optical markers for acquisition of groundtruth camera trajectories. All image sequences were resizedto 640× 480 for tests.

Fig. 4. The indoor environment for experiments is shown on the left, andthe data collection device equipped with the camera GoPro, is presented onthe right.

TABLE ITABLE I: LOCALIZATION PERFORMANCE. RPE (0.1 M) AND APE (M)

Seq. ORB-SLAM Point-only TextSLAMRPE APE RPE APE RPE APE

Indoor 01 0.342 0.161 0.192 0.223 0.188 0.196Indoor 02 0.300 0.150 0.189 0.164 0.187 0.150Indoor 03 0.228 0.144 0.207 0.166 0.209 0.155Indoor 04 0.188 0.145 – – 0.173 0.171Indoor 05 0.277 0.120 0.172 0.160 0.174 0.161

The bar ’–’ indicates the algorithm fails to finish the whole trajectory.

Fig. 5. TextSLAM is robust to blurry images caused by rapid cameramotions. The estimated 3D text map and camera trajectory of TextSLAMare shown on the left. By contrast, point-only method failed to track featurepoints on severely blurry images as shown on the right.

B. Indoor scene with ground truth

The indoor environment is shown in Fig. 4, which isequipped with a motion capture system to obtain the groundtruth camera trajectories of millimeter accuracy within anarea of 4m × 4m. We placed random texts in the room.Three methods were evaluated : our text-based method, ourpoint-only method, and ORB-SLAM [4] where loop closingwas disabled for fair comparison.

a) Trajectory estimation: We present both the relativepose error (RPE) and the absolute pose error (APE) in Tab.I. Note that the errors of those methods are very close toeach other. The reason should be the test scene is verysmall and highly textured. In such cases, using only featurepoints should work well. ORB-SLAM slightly outperformsboth of our methods, which is not surprising because ORB-SLAM adopts sophisticated map reuse mechanism basedon covisibility pose graph. Our methods currently are infact odometry systems. Nevertheless, we still observe aperformance gain in using text features compared with ourpoint-only implementation.

We also evaluate the robustness of the proposed methodunder fast camera motion. The rapid motion caused severeimage blur as shown in Fig. 5, hence made our point-onlymethod fail in all cases. ORB-SLAM failed only at onetest. This is also because its well implemented relocalizationmechanism. By contrast, our text-based method works wellin those tests and performs much more accurate as shown inTab. II. This is largely due to our direct approach towardstext objects using photometric errors.

b) 3D text maps: We use the angular error of eachestimated text plane and visually inspection to evaluatethe mapping performance. The angular error measures thedifference between the normal of the estimated text plane andthe ground truth, namely α = arccos(|nT

t ngt|/‖nt‖‖ngt‖).The ground truth normal was acquired by a few opticalmarkers on the text plane as shown in Fig. 4. Since no

TABLE IITABLE II: INDOOR RAPID PERFORMANCE. RPE (0.1 M) AND APE (M)

Seq. ORB-SLAM Point-only TextSLAMRPE APE RPE APE RPE APE

Rapid 01 – – – – 0.271 0.029Rapid 02 0.367 0.063 – – 0.322 0.031Rapid 03 0.338 0.061 – – 0.212 0.022

The bar ’–’ indicates the algorithm fails to finish the whole trajectory.

visual SLAM system that can directly generate text maps isavailable for tests, we implemented a loosely-coupled systembased on ORB-SLAM for comparison by fitting the textplane to the 3D text points using three-point RANSAC.

The statistic of angular errors are presented in Fig. 8 andone of the test results are shown in Fig. 7. The results showthat the 3D text map produced by TextSLAM is substantiallybetter than that of the plane fitting approach based on ORB-SLAM. The reason is that the point clouds generated byORB-SLAM are in fact noisy as shown in Fig. 7. We care-fully checked what causes those noisy points and found thatsome of them were caused by wrong feature correspondencesbecause of repetitive text patterns, while some of them hadno visually wrong feature correspondence, which we suspectmay be caused by inaccurate feature extraction.

C. Real-word tests

We tested the proposed method in a commercial center asshown in the first column in Fig. 6. Because it is difficultto acquire the ground truth for either the camera trajectoryor the text objects in such real world scenes, we presentonly visual results to show the efficacy of our method. Thesecond and three columns in Fig. 6 illustrate reconstructed3D text labels and the estimate trajectories. A side-by-sidecomparison with ORB-SLAM from the top-down view isalso presented on the last column. Though no ground truthis available, we can still observe the noisy point clouds inthe ORB-SLAM results. As highlighted by rectangles inFig. 6, the text maps better reveal the planar structures ofthe scene than the noisy point clouds from ORB-SLAM.As aforementioned, those noisy points may be caused byeither incorrect feature correspondences or inaccurate featurelocation in the images. No matter what causes, those noisypoints prevent us from acquiring accurate 3D text maps.

VI. CONCLUSION AND FUTURE WORK

We present a visual SLAM method that tightly coupledwith the novel text features. Experiments have been con-ducted in both artificial indoor cases with ground truthand real world scenes. The results show that our text-based SLAM method performs better than point-only SLAMmethod does, especially in blurry video sequences thatcaused by fast camera motions. However, the localizationperformance gain is not as large as we expected in the theindoor tests, even when point-based methods generate verynoisy point clouds. This could be that camera pose estimationis less sensitive to noisy points because robust approachesare usually involved. Nevertheless, the results show that our

Fig. 6. Real word tests in a shopping center.The text detection results are shown in the first column. Three typical locations are shown and enlarged ineach row. The second and third columns show the close-up and top-down views. For comparison, the ORB performance of the same locations are alsopresented in the third column.We can observe the noisy point clouds visually, as enclosed in red rectangles.

A

A B C

B

C

A

B

C

TextSLAM

ORB-SLAM

Fig. 7. Though RANSAC was adopted, plane fitting on the point cloudsfrom ORB-SLAM still produced noisy results (as shown in the bottom row).By contrast, TextSLAM avoids such problem by matching or tracking a textobject as a whole using photometric errors to estimate the 3D text patchesaccurately (as shown in the top row).

9 18 27 36 45 54 63 72 81 90

degree (°)

0

0.1

0.2

0.3

0.4

0.5

TextSLAM

ORB-SLAM

Fig. 8. The statistic distribution of the angular errors. The results ofTextSLAM and ORB-SLAM are illustrated in red and blue, respectively.

method generates much more accurate 3D text maps thandoes the loosely-coupled method that based on the state-of-the-art visual SLAM system. Future work includes acquiringreal world datasets with highly accurate ground truth andalso incorporate text-based loop closing in our tighly-coupledmethod.

REFERENCES

[1] L. Heng, D. Honegger, G. H. Lee, L. Meier, P. Tanskanen, F. Fraundor-fer, and M. Pollefeys, “Autonomous visual mapping and explorationwith a micro aerial vehicle,” Journal of Field Robotics, vol. 31, no. 4,pp. 654–675, 2014.

[2] H. Lategahn, A. Geiger, and B. Kitt, “Visual slam for autonomousground vehicles,” in 2011 IEEE International Conference on Roboticsand Automation. IEEE, 2011, pp. 1732–1737.

[3] D. Chekhlov, A. P. Gee, A. Calway, and W. Mayol-Cuevas, “Ninja ona plane: Automatic discovery of physical planes for augmented realityusing visual slam,” in Proceedings of the 2007 6th IEEE and ACMInternational Symposium on Mixed and Augmented Reality. IEEEComputer Society, 2007, pp. 1–4.

[4] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “Orb-slam: aversatile and accurate monocular slam system,” IEEE Transactionson Robotics, vol. 31, no. 5, pp. 1147–1163, 2015.

[5] A. J. Davison, “Real-time simultaneous localisation and mapping witha single camera,” in null. IEEE, 2003, p. 1403.

[6] C. Forster, M. Pizzoli, and D. Scaramuzza, “Svo: Fast semi-directmonocular visual odometry,” in Robotics and Automation (ICRA),2014 IEEE International Conference on. IEEE, 2014, pp. 15–22.

[7] J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” IEEEtransactions on pattern analysis and machine intelligence, vol. 40,no. 3, pp. 611–625, 2018.

[8] H. Zhou, D. Zou, L. Pei, R. Ying, P. Liu, and W. Yu, “Structslam:Visual slam with building structure lines,” IEEE Transactions onVehicular Technology, vol. 64, no. 4, pp. 1364–1375, 2015.

[9] A. J. Trevor, J. G. Rogers, and H. I. Christensen, “Planar surface slamwith 3d and 2d sensors,” in Robotics and Automation (ICRA), 2012IEEE International Conference on. IEEE, 2012, pp. 3041–3048.

[10] Y. Zhu, C. Yao, and X. Bai, “Scene text detection and recognition:Recent advances and future trends,” Frontiers of Computer Science,vol. 10, no. 1, pp. 19–36, 2016.

[11] X.-C. Yin, Z.-Y. Zuo, S. Tian, and C.-L. Liu, “Text detection, trackingand recognition in video: a comprehensive survey,” IEEE Transactionson Image Processing, vol. 25, no. 6, pp. 2752–2773, 2016.

[12] A. Veit, T. Matera, L. Neumann, J. Matas, and S. Belongie, “Coco-text:Dataset and benchmark for text detection and recognition in naturalimages,” arXiv preprint arXiv:1601.07140, 2016.

[13] M. Iwamura, T. Matsuda, N. Morimoto, H. Sato, Y. Ikeda, and K. Kise,

“Downtown osaka scene text dataset,” in European Conference onComputer Vision. Springer, 2016, pp. 440–455.

[14] D. Karatzas, L. Gomez-Bigorda, A. Nicolaou, S. Ghosh, A. Bagdanov,M. Iwamura, J. Matas, L. Neumann, V. R. Chandrasekhar, S. Lu, et al.,“Icdar 2015 competition on robust reading,” in Document Analysis andRecognition (ICDAR), 2015 13th International Conference on. IEEE,2015, pp. 1156–1160.

[15] X. Rong, B. Li, J. P. Munoz, J. Xiao, A. Arditi, and Y. Tian,“Guided text spotting for assistive blind navigation in unfamiliar in-door environments,” in International Symposium on Visual Computing.Springer, 2016, pp. 11–22.

[16] B. Li, J. P. Munoz, X. Rong, Q. Chen, J. Xiao, Y. Tian, A. Arditi,and M. Yousuf, “Vision-based mobile indoor assistive navigation aidfor blind people,” IEEE Transactions on Mobile Computing, vol. 18,no. 3, pp. 702–714, 2019.

[17] H.-C. Wang, C. Finn, L. Paull, M. Kaess, R. Rosenholtz, S. Teller, andJ. Leonard, “Bridging text spotting and slam with junction features,”2015.

[18] A. P. Gee, D. Chekhlov, W. W. Mayol-Cuevas, and A. Calway,“Discovering planes and collapsing the state space in visual slam.”in BMVC, 2007, pp. 1–10.

[19] A. P. Gee, D. Chekhlov, A. Calway, and W. Mayol-Cuevas, “Dis-covering higher level structure in visual slam,” IEEE Transactions onRobotics, vol. 24, no. 5, pp. 980–990, 2008.

[20] M. Y. Yang and W. Forstner, “Plane detection in point cloud data,” inProceedings of the 2nd int conf on machine control guidance, Bonn,vol. 1, 2010, pp. 95–104.

[21] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers,“A benchmark for the evaluation of rgb-d slam systems,” in 2012IEEE/RSJ International Conference on Intelligent Robots and Systems.IEEE, 2012, pp. 573–580.

[22] P. Kim, B. Coltin, and H. Jin Kim, “Linear rgb-d slam for planar envi-ronments,” in Proceedings of the European Conference on ComputerVision (ECCV), 2018, pp. 333–348.

[23] N. Molton, A. J. Davison, and I. D. Reid, “Locally planar patchfeatures for real-time structure from motion.” in Bmvc. Citeseer,2004, pp. 1–10.

[24] R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. Kelly, andA. J. Davison, “Slam++: Simultaneous localisation and mapping at thelevel of objects,” in Proceedings of the IEEE conference on computervision and pattern recognition, 2013, pp. 1352–1359.

[25] D. Galvez-Lopez, M. Salas, J. D. Tardos, and J. Montiel, “Real-timemonocular object slam,” Robotics and Autonomous Systems, vol. 75,pp. 435–449, 2016.

[26] J. Civera, D. Galvez-Lopez, L. Riazuelo, J. D. Tardos, and J. Mon-tiel, “Towards semantic slam using a monocular camera,” in 2011IEEE/RSJ International Conference on Intelligent Robots and Systems.IEEE, 2011, pp. 1277–1284.

[27] J. McCormac, R. Clark, M. Bloesch, A. Davison, and S. Leuteneg-ger, “Fusion++: Volumetric object-level slam,” in 2018 InternationalConference on 3D Vision (3DV). IEEE, 2018, pp. 32–41.

[28] S. Yang and S. Scherer, “Cubeslam: Monocular 3-d object slam,” IEEETransactions on Robotics, 2019.

[29] ——, “Monocular object and plane slam in structured environments,”IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 3145–3152,2019.

[30] L. Nicholson, M. Milford, and N. Sunderhauf, “Quadricslam: Dualquadrics from object detections as landmarks in object-oriented slam,”IEEE Robotics and Automation Letters, vol. 4, no. 1, pp. 1–8, 2018.

[31] S. Houben, D. Droeschel, and S. Behnke, “Joint 3d laser and visualfiducial marker based slam for a micro aerial vehicle,” in MultisensorFusion and Integration for Intelligent Systems (MFI), 2016 IEEEInternational Conference on. IEEE, 2016, pp. 609–614.

[32] R. Tapu, B. Mocanu, and T. Zaharia, “A computer vision-based percep-tion system for visually impaired,” Multimedia Tools and Applications,vol. 76, no. 9, pp. 11 771–11 807, 2017.

[33] M. Tomono and S. Yuta, “Mobile robot navigation in indoor envi-ronments using object and character recognition,” in Robotics andAutomation, 2000. Proceedings. ICRA’00. IEEE International Con-ference on, vol. 1. IEEE, 2000, pp. 313–320.

[34] C. Case, B. Suresh, A. Coates, and A. Y. Ng, “Autonomous signreading for semantic mapping,” in Robotics and Automation (ICRA),2011 IEEE International Conference on. IEEE, 2011, pp. 3297–3303.

[35] X. Zhou, C. Yao, H. Wen, Y. Wang, S. Zhou, W. He, and J. Liang,“East: an efficient and accurate scene text detector,” in 2017 IEEE

Conference on Computer Vision and Pattern Recognition (CVPR).IEEE, 2017, pp. 2642–2651.

[36] E. Rosten and T. Drummond, “Machine learning for high-speed cornerdetection,” in European conference on computer vision. Springer,2006, pp. 430–443.

[37] J. Shi and C. Tomasi, “Good features to track,” Cornell University,Tech. Rep., 1993.