CC BY
27
2018, Т. 160, кн. 2 С. 300-308
УЧЕНЫЕ ЗАПИСКИ КАЗАНСКОГО УНИВЕРСИТЕТА. СЕРИЯ ФИЗИКО-МАТЕМАТИЧЕСКИЕ НАУКИ
ISSN 2541-7746 (Print) ISSN 2500-2198 (Online)
UDK 519.23
alignment of vector fields on manifolds via contraction mappings
O.N. Kachana, Yu.A. Yanovicha'b'c, E.N. Abramovc
aSkolkovo Institute of Science and Technology, Moscow, 143026 Russia bKharkevich Institute for Information Transmission Problems Russian Academy of Sciences, Moscow, 127051 Russia cNational Research University Higher School of Economics, Moscow, 101000 Russia
Abstract
According to the manifold hypothesis, high-dimensional data can be viewed and meaningfully represented as a lower-dimensional manifold embedded in a higher dimensional feature space. Manifold learning is a part of machine learning where an intrinsic data representation is uncovered based on the manifold hypothesis.
Many manifold learning algorithms were developed. The one called Grassmann & Stiefel eigenmaps (GSE) has been considered in the paper. One of the GSE subproblems is tangent space alignment. The original solution to this problem has been formulated as a generalized eigenvalue problem. In this formulation, it is plagued with numerical instability, resulting in suboptimal solutions to the subproblem and manifold reconstruction problem in general.
We have proposed an iterative algorithm to directly solve the tangent spaces alignment problem. As a result, we have obtained a significant gain in algorithm efficiency and time complexity. We have compared the performance of our method on various model data sets to show that our solution is on par with the approach to vector fields alignment formulated as an optimization on the Stiefel group.
Keywords: manifold learning, dimensionality reduction, numerical optimization, vector field estimation
Introduction
In the last few decades, advances in computational power and data acquisition techniques have led to the data explosion. Notably, not only the amount of data increased, but also the dimensionality of these data [1].
Nowadays, one should be able to analyze data coming from areas of computer vision, 3D-sensing, medical image analysis, videos, sound, speech, sensor networks, user preferences in recommendation systems, DNA microarrays to name a few.
Generalization of machine learning algorithms in high-dimensional spaces is hard because of the curse of dimensionality [2]. One way to overcome it is the manifold assumption that data in high-dimensional spaces lie on or near a low-dimensional manifold due to correlation and sparsity.
Indeed, most high-dimensional data are non-Euclidean, manifold-valued:
• shape data [3], especially in biomedical imaging [4];
• human poses and hand gestures [5, 6];
• images of an object under different lighting conditions, rigid bodies, under rotation and affine transformations [7];
• observation and action spaces in robotics [8, 9] and reinforcement learning problems [10].
Ideally, machine learning algorithms should take into account an implicit manifold nature of such data into account instead of naively viewing and representing complex data as points in the Euclidean space.
1. Overview
1.1. Manifold learning. Manifold learning [7] is a comparatively new approach to dimensionality reduction dating back to the early 2000s. It takes into account the geometrical properties of data and can explain why neural networks work - they intrinsically learn data manifold [11].
1.2. Manifold learning problem. Given Xn = {x1, x2,..., xn} ~ M C RD .
let us find a mapping f : RD ^ Rd, where d ^ D and M is a manifold embedded in RD .
A large number of manifold learning algorithms exists, including Laplacian eigen-maps (LE) [12], locally linear embedding (LLE) [13], and locally tangent space alignment (LTSA) [14]. These algorithms differ in a sense of which geometric properties of the underlying manifold they aim to preserve.
1.3. Grassmann & Stiefel eigenmaps. An amplification of manifold learning has been proposed recently [15] and the Grassmann & Stiefel eigenmaps (GSE) algorithm to solve its problem has been introduced. This novel approach called tangent bundle manifold learning (TBML) requires proximity not only between the data manifold and the reconstructed manifold, but also between their tangent spaces. The authors theoretically justify that this improves the generalization ability and better preserves local structure of a manifold [15]. Within the method, these results of treatment of the tangent bundle as Grassmann manifold GD,d and in manifold approximation of Stiefel manifold SDtd naturally arise in solving the (generalized) eigenvalue problem. Some details of the GSE solution are considered in [16, 17], and partial statistical analysis is provided in [18, 19].
The Grassmann & Stiefel eigenmaps algorithm includes the following steps [20].
1. Local estimation of tangent spaces
For each point x. G Xn of dimension D, let us construct a neighborhood set U(x.), whether by choosing points within the e-neighborhood based on Euclidean distance U(x.) = |x' G Xn : ||x' — Xj||2 < e} or by identifying the k-nearest neighbors. In this work, the method of k-nearest neighbors is considered.
Let us estimate a tangent space of dimension d to each point xj by applying the principal component analysis to find a set of basis vectors Qpca(x.) = |qi(xj),..., qd(x.)}T corresponding to d largest eigenvalues of the correlation matrix.
2. Tangent spaces alignment
Let us find a new set of bases, smoothly changing from point to point, such that the span of the new basis at each point is in the span of the basis estimated with PCA:
]T K(xj,xj) • ||H(xj) — H(x
■j) HF
-> mm
. . . {HCxi}}^!
j,3 = 1 1
s.t. Span(H(xj)) = Span(QpCA(xj)),
where Span(A) is a linear hull of q columns ai,..., aq of the matrix A = (ai,..., aq).
The authors of the GSE algorithm showed [15] that this problem can be formulated and solved in terms of linear algebra as a generalized eigenvalue problem.
Let V(x.) = QpcA(xj)TH(x.), then
$V = AAV.
2
Despite being mathematically beautiful, numerical solution of this problem is unstable. As we need to estimate a set of change of basis matrices, a small relative average
error for the whole set "V does not imply a small relative error for all Vi. Small norms have large relative errors and vice versa. Another problem is the computational complexity of matrix inverse requiring O(n3) time.
Tangent spaces alignment is the key problem of the GSE algorithm. Solving this problem efficiently is crucial for construction of a point embedding in the next step.
3. Construction of embedding and reconstruction mappings
At this stage, embedding and reconstruction mappings based on the aligned tangent spaces are constructed. This stage is out of the scope of this work and , therefore, we will not describe it in detail.
2. Tangent spaces alignment problem
2.1. Problem statement. Instead of solving the eigenvalue problem, we return to the original problem and solve it directly, enforcing the norm of tangent spaces bases with the additional orthonormality constraint
n
(xj,Xj) • ||H(xj) - H(xj)||F - rnin , (1)
i,j = 1 |H(xi)|i=1
s.t. Span(H(xj)) = Span(QpCA(xj)), H(xj)T H(xj)= Id.
2.2. Solution. The proposed optimization scheme is iterative, the bases depend on the ones computed in the previous iteration:
H(m+1) = f (H(!m),..., Hnm)).
The scheme consists of three steps, performed on each iteration
1) update Hi via the Nadaraya-Watson nonparametric regression [21] for each point
xi ^ Xn ;
2) projecting updated basis vectors back to tangent space of a point;
3) normalization of a set of basis vectors.
Let us describe steps in more detail.
The Nadaraya-Watson nonparametric regression depends on kernel selection, which defines the measure of similarity between points within the neighborhood of the point xi. Numerous different kernels exist. We use the k-nearest neighbors kernel with the number of neighbors k equal to the number of neighbors used to construct a neighborhood set U(x) and at first denote:
J2K(xi, xj) • Hjm)
H(m+1)
J2K (xi, xj) j
After averaging a set of basis vectors within the neighborhood, we obtained set Hf+1) ^M. So, we do a projection back to the tangent space using the projection d x d matrix n(xi):
n(xi) = QpcA(xi) • QpcA(xi)T.
The second step is
Y^K(xi, xj) • Hjm)
H(m+1) = QiQT • j
J2K (xi, xj)
—I— Contraction mapping —|— Optimization on stiefel group
200 400 800
log(sample size)
200 400 800
log(sample size)
a) Cost function b) Computation time
Fig. 1. Cylinder data set
After the projection to tangent space TXiM, the norm of the set of basis vectors changes. So, we must perform a correction, considering strict orthonormality
HT Hj = VT qT QjVj = Id.
We ensure it by carrying out SVD-decomposition [22] and replacing X with Id at the last step of each iteration
H
(m+1)
J2K(xj, x.) • Hjj
QiQT • -
(m} j
U£VJ
J2K (xj, x.) j
(m+1)
H
H(m+1} := UIdVT.
The computational complexity of the proposed solution for the tangent spaces alignment problem was improved:
• full matrix: O(n2) instead of O(n3) for solving the generalized eigenvalue problem;
• sparse matrix: O(nk) instead of O(n2k), where k is a sparsity parameter (the number of neighbors).
3. Experiments
The half of two-dimensional sphere §21>o C R3 and the cylinder S1 x [0,1]) are considered as manifold M . All samples are iid uniformly distributed.
The algorithm code and experiment scripts were made in Python. The transformed and scaled version of functional (1)
2
A = — ndrk^jj •Tr (VT (qT Qj )Vj
^ mm
(2)
and the computation time were calculated for different sample sizes and number of neighbors k (used to construct kernel K(x., x.)). For two close points x., x. function
- Tr (HT Hj) ^ 1, therefore functional A should be close to —2. The value of functional
1.94
-1.95
□ -1.96
1.97
-1.
1.99
0
a) Cost function b) Computation time
Fig. 2. Sphere data set
-,,-,1.11,, b) Sphere data set
a) Cylinder data set
Fig. 3. Aligned tangent spaces
A as a function of sample size and its calculation time were estimated for sample sizes n = 100 • 2l, l = 0, . . . , 4 for cylinder and two-dimensional sphere. For each sample size, 10 samples were calculated.
The results of the proposed algorithm and the first-order optimization algorithm from [23] were compared. Also, the initialization from [23] was used to get the same neighborhoods of points and initial PCA-estimation of tangent spaces orientation to measure the algorithms performance under the same conditions. In Figs. 1 and 2, the means of functional A and computation time +/— their standard deviations are depicted.
Version (2) for the proposed approach is comparable with that in [23] (better or equal for small samples and tends to be a bit worse for larger ones), while the computation time grows slower with the sample size increase.
Visually, the optimization results for both algorithms look similar and smooth. So, only the optimized vector fields obtained by the contraction mapping method are shown in Fig. 3.
Conclusions
An iterative optimization scheme for vector fields alignment on manifolds was proposed and implemented. A computational experiment with artificial data was performed.
Its results prove the computational quality and efficiency of the approach. The vector field alignment problem is a part of the Grassmann & Stiefel eigenmaps manifold learning algorithm, and the findings of this paper can be used to improve its computational efficiency.
Acknowledgements. The study was supported by the Russian Science Foundation (project no. 14-50-00150).
References
1. Giraud Ch. Introduction to High-Dimensional Statistics. New York, Chapman and Hall/CRC, 2014. 270 p.
2. Donoho D.L. High-dimensional data analysis: The curses and blessings of dimensionality. AMS Conf. on Math Challenges of 21st Century, 2000, pp. 1-31.
3. Ezuz D., Solomon J., Kim V.G., Ben-Chen M. GWCNN: A metric alignment layer for deep shape analysis. Comput. Graphics Forum, 2017, vol. 36, no. 5, pp. 49-57. doi: 10.1111/cgf.13244.
4. Qiu A., Lee A., Tan M., Chung M.K. Manifold learning on brain functional networks in aging. Med. Image Anal., 2015, vol. 20, no. 1, pp. 52-60. doi: 10.1016/j.media.2014.10.006.
5. Bronstein M.M., Bruna J., LeCun Y., Szlam A., Vandergheynst P. Geometric deep learning: Going beyond Euclidean data. IEEE Signal Process. Mag., 2017, vol. 34, no. 4, pp. 18-42. doi: 10.1109/MSP.2017.2693418.
6. Xu C., Govindarajan L.N., Zhang Y., Cheng L., Lie-X: Depth image based articulated object pose estimation, tracking, and action recognition on lie groups. Int. J. Comput. Vision, 2017, vol. 123, no. 3, pp. 454-478. doi: 10.1007/s11263-017-0998-6.
7. Seung H.S., Lee D.D. COognition. The manifold ways of perception. Science, 2000, vol. 290, no. 5500, pp. 2268-2269. doi: 10.1126/science.290.5500.2268.
8. Zeestraten M.J.A., Havoutis I., Silverio J., Calinon S., Caldwell D.G. An approach for imitation learning on riemannian manifolds. IEEE Rob. and Autom. Lett., 2017, vol. 2, no. 3, pp. 1240-1247. doi: 10.1109/LRA.2017.2657001.
9. Klingensmith M., Koval M.C., Srinivasa S.S., Pollard N.S., Kaess M. The manifold particle filter for state estimation on high-dimensional implicit manifolds. 2017 IEEE Int. Conf. on Robotics and Automation (ICRA). Singapore, 2017, pp. 4673-4680. doi: 10.1109/ICRA.2017.7989543.
10. Bush K., Pineau J. Manifold embeddings for model-based reinforcement learning under partial observability. In: Bengio Y., Schuurmans D., Lafferty J.D., Williams C.K.I., Culotta A. (Eds.)Advances in Neural Information Processing Systems. Curran Associates, Inc., 2009, pp. 189-197.
11. Brahma P., Wu D., She Y., Why deep learning works: A manifold disentanglement perspective. IEEE Trans. Neural Networks and Learn. Syst., 2016, vol. 27, no. 10, pp. 19972008. doi: 10.1109/TNNLS.2015.2496947.
12. Belkin M., Niyogi P. Laplacian Eigenmaps for dimensionality reduction and data representation. Neural Comput., 2003, vol. 15, no. 6, pp. 1373-1396. doi: 10.1162/089976603321780317.
13. Roweis S.T., Saul L.K. Nonlinear dimensionality reduction by locally linear embedding. Science, 2000, vol. 290, no. 5500, pp. 2323-2326. doi: 10.1126/science.290.5500.2323.
14. Zhang Z., Zha H. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM J. Sci. Comput., 2004, vol. 26, no. 1, p. 313-338. doi: 10.1137/S1064827502419154.
15. Bernstein A., Kuleshov A.P. Manifold learning: Generalizing ability and tangent proximity. Int. J. Software Inf., 2013, vol. 7, no. 3, pp. 359-390.
16. Bernstein A., Kuleshov A., Yanovich Y., Manifold learning in regression tasks. In: Lecture Notes in Computer Science, 2015, vol. 9047, pp. 414-423.
17. Bernstein A.V., Kuleshov A.P., Yanovich Yu.A. Locally isometric and conformal parameterization of image manifold. Proc. 8th Int. Conf. on Machine Vision. Verikas A., Radeva P., Nikolaev D. (Eds.). Vol. 987507. Barcelona, 2015, pp. 1-7. doi: 10.1117/12.2228741.
18. Yanovich Yu. Asymptotic properties of local sampling on manifold. J. Math. Stat., 2016, vol. 12, no. 3, pp. 157-175. doi: 10.3844/jmssp.2016.157.175.
19. Yanovich Yu. Asymptotic properties of nonparametric estimation on manifold. Proc. 6th Workshop on Conformal and Probabilistic Prediction and Applications, PMLR, 2017, vol. 60, pp. 18-38.
20. Bernstein A., Burnaev E., Erofeev P. Comparative study of nonlinear methods for manifold learning. Proc. Conf. "Information Technologies and Systems", 2012, pp. 85-91.
21. Bishop C.M. Pattern Recognition and Machine Learning. New York, Springer, 2006. xx, 738 p.
22. Golub G.H., van Loan Ch.F. Matrix Computations. Baltimore, Johns Hopkins Univ. Press, 1996. 694 p.
23. Abramov E., Yanovich Yu. Smooth vector fields estimation on manifolds by optimization on Stiefel group. Uchenye Zapiski Kazanskogo Universiteta. Seriya Fiziko-Matematicheskie Nauki, 2018, vol. 160, no. 2, pp. 220-228.
Received October 11, 2017
Kachan Oleg Nikolaevich, PhD Student of the Center for Computational and DataIntensive Science and Engineering
Skolkovo Institute of Science and Technology
ul. Nobelya, 3, Territory of the Innovation Center "Skolkovo", Moscow, 143026 Russia E-mail: oleg.kachan@skoltech.ru
Yanovich Yury Alexandrovich, Candidate of Physical and Mathematical Sciences, Researcher of the Center for Computational and Data-Intensive Science and Engineering; Researcher of the Intelligent Data Analysis and Predictive Modeling Laboratory; Lecturer of the Faculty of Computer Science
Skolkovo Institute of Science and Technology
ul. Nobelya, 3, Territory of the Innovation Center "Skolkovo", Moscow, 143026 Russia Kharkevich Institute for Information Transmission Problems, Russian Academy of Sciences
Bolshoy Karetny pereulok 19, str. 1, Moscow, 127051 Russia National Research University "Higher School of Economics"
ul. Myasnitskaya, 20, Moscow, 101000 Russia E-mail: yury.yanovich@iitp.ru
Abramov Evgeny Nikolayevich, graduate student of Faculty of Computer Science National Research University "Higher School of Economics"
ul. Myasnitskaya, 20, Moscow, 101000 Russia E-mail: petzchner@gmail.com
УДК 519.23
Выравнивание векторных полей на многообразиях методом сжимающих отображений
О.Н. Качан1, Ю.А. Янович1'3'2 , Е.Н. Абрамов3
1 Сколковский институт науки и технологий, г. Москва, 143026, Россия
2Институт проблем передачи информации Харкевича РАН, г. Москва, 127051, Россия 3Национальный исследовательский университет «Высшая школа экономики», г. Москва, 101000, Россия
Аннотация
Многие задачи анализа данных связаны с высокоразмерными данными, и феномен «проклятия размерности» является препятствием для использования целого ряда методов, чтобы их решить. Часто в приложениях многомерные данные занимают лишь очень малую часть высокоразмерного пространства наблюдений, имеющую существенно меньшую размерность по сравнению с размерностью этого пространства. Модель многообразия для таких данных, в соответствии с которой данные лежат на (или вблизи) неизвестного низкоразмерного многообразия данных, вложенного в охватывающее высокоразмерное пространство, является популярной моделью для таких данных. Задачи анализа данных, решаемых в рамках этой модели, принято называть задачами моделирования многообразий, общая цель которых состоит в выявлении низкоразмерной структуры в лежащих на многообразии данных по имеющейся конечной выборке. Если точки выборки извлечены из многообразия в соответствии с неизвестной вероятностной мерой на многообразии данных, мы имеем дело со статистическими задачами на многообразии данных. Статья содержит обзор таких статистических задач и методов их решения.
Ключевые слова: анализ данных, математическая статистика, моделирование многообразий, оценка плотности на многообразиях, регрессия на многообразиях
Поступила в редакцию 11.10.17
Качан Олег Николаевич, аспирант Центра по научным и инженерным вычислительным технологиям для задач с большими массивами данных Сколковский институт науки и технологий
ул. Нобеля, д. 3, Территория Инновационного Центра «Сколково», г. Москва, 143026, Россия
E-mail: oleg.kachan@skoltech.ru
Янович Юрий Александрович, кандидат физико-математических наук, научный сотрудник Центра по научным и инженерным вычислительным технологиям для задач с большими массивами данных; научный сотрудник лаборатории интеллектуального анализа данных и предсказательного моделирования; старший преподаватель факультета компьютерных наук
Сколковский институт науки и технологий
ул. Нобеля, д. 3, Территория Инновационного Центра «Сколково», г. Москва, 143026, Россия
Институт проблем передачи информации им. А.А. Харкевича РАН
Большой Каретный переулок, д. 19, стр. 1, г. Москва, 127051, Россия Национальный исследовательский университет «Высшая школа экономики»
ул. Мясницкая, д. 20, г. Москва, 101000, Россия E-mail: yury.yanovich@iitp.ru
Абрамов Евгений Николаевич, студент факультета компьютерных наук Национальный исследовательский университет «Высшая школа экономики»
ул. Мясницкая, д. 20, г. Москва, 101000, Россия E-mail: petzchner@gmail.com
I For citation: Kachan O.N., Yanovich Yu.A., Abramov E.N. Alignment of vector fields ( on manifolds via contraction mappings. Uchenye Zapiski Kazanskogo Universiteta. Seriya \ Fiziko-Matematicheskie Nauki, 2018, vol. 160, no. 2, pp. 300-308.
/ Для цитирования: Kachan O.N., Yanovich Yu.A., Abramov E.N. Alignment of vector ( fields on manifolds via contraction mappings // Учен. зап. Казан. ун-та. Сер. Физ.-ма-\ тем. науки. - 2018. - Т. 160, кн. 2. - С. 300-308.
