Supervised machine learning with Delaunay triangulation learner and collaborative gradient boosting

Abstract

In this thesis, we discuss some statistical machine learning methods and their ensemble learning approachs with theories and applications.

Firstly, we propose a new piecewise linear learner, called the Delaunay triangulation learner (DTL). Based on the data samples in a p-dimensional feature space, the Delaunay triangulation algorithm provides a unique triangulation of the space. The triangulation separates the convex hull of the samples into a series of disjoint p-simplices, where the samples are the vertices of the p-simplices. The DTL is constructed by fitting the responses through linear interpolation functions on each of the Delaunay simplices, and thus it approximates the whole functional by a piecewise linear function. In its ensemble learning approaches, we propose the bagging DTLs, random crystal and the boosting DTL, where the DTLs are constructed on the subspaces of the features, and the feature interactions are captured by Delaunay triangle meshes. Extensive numerical studies are conducted to compare the proposed DTL and its ensembles with tree-based counterparts. The DTL methods show competitive performances in various settings, and particularly for smooth functionals the DTL demonstrates its superiority over others.

Secondly, we propose a collaborative gradient boosting (CGB) algorithm, which can leverage the full potential of boosting multiple types of base learners by using the intrinsic regularization. Inspired by the idea of coopetition (i.e., cooperation and competition), we allow the base learners of different types to collaborate with as well as competing against each other in gradient descent steps. In the CGB algorithm, different types of base learners typically learn from each other toward better model accuracy, i.e., poor-performing base learners would approach better ones. Extensive numerical studies show that the CGB dramatically improves the prediction accuracy by utilizing different types of learners. Our method yields a general strategy that performs better on real data from a wide range of domains.

Finally, we introduce a new concept of the average holding price (AHP) in stock market. We show that, under certain assumptions on the investors’ behavior, the average holding price of a stock can be estimated, based on the historical trading prices and volumes. In contrast to the moving average of the stock prices, the AHP can serve as a benchmark for estimating the average profit/loss level of the stock holders. The numerical algorithm for the AHP depends on an recursive equation, which enables us to compute the real-time AHP from any time point of the stock trading. As illustrated in the examples, some trading strategies can also be built upon the AHP.