File Download

There are no files associated with this item.

  Links for fulltext
     (May Require Subscription)
Supplementary

Article: A Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With Applications

TitleA Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With Applications
Authors
KeywordsDamped parameter
limited memory BFGS (LBFGS)
nonconjugate exponential models
nonconvex optimization
stochastic quasi-Newton (SQN) method
Issue Date2020
PublisherInstitute of Electrical and Electronics Engineers. The Journal's web site is located at http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=72
Citation
IEEE Transactions on Neural Networks and Learning Systems, 2020, v. 31 n. 11, p. 4776-4790 How to Cite?
AbstractEnsuring the positive definiteness and avoiding ill conditioning of the Hessian update in the stochastic Broyden-Fletcher-Goldfarb-Shanno (BFGS) method are significant in solving nonconvex problems. This article proposes a novel stochastic version of a damped and regularized BFGS method for addressing the above problems. While the proposed regularized strategy helps to prevent the BFGS matrix from being close to singularity, the new damped parameter further ensures the positivity of the product of correction pairs. To alleviate the computational cost of the stochastic limited memory BFGS (LBFGS) updates and to improve its robustness, the curvature information is updated using the averaged iterate at spaced intervals. The effectiveness of the proposed method is evaluated through the logistic regression and Bayesian logistic regression problems in machine learning. Numerical experiments are conducted by using both synthetic data set and several real data sets. The results show that the proposed method generally outperforms the stochastic damped LBFGS (SdLBFGS) method. In particular, for problems with small sample sizes, our method has shown superior performance and is capable of mitigating ill-conditioned problems. Furthermore, our method is more robust to the variations of the batch size and memory size than the SdLBFGS method.
Persistent Identifierhttp://hdl.handle.net/10722/294070
ISSN
2023 Impact Factor: 10.2
2023 SCImago Journal Rankings: 4.170
ISI Accession Number ID

 

DC FieldValueLanguage
dc.contributor.authorCHEN, H-
dc.contributor.authorWu, HC-
dc.contributor.authorChan, SC-
dc.contributor.authorLam, WH-
dc.date.accessioned2020-11-23T08:25:54Z-
dc.date.available2020-11-23T08:25:54Z-
dc.date.issued2020-
dc.identifier.citationIEEE Transactions on Neural Networks and Learning Systems, 2020, v. 31 n. 11, p. 4776-4790-
dc.identifier.issn2162-237X-
dc.identifier.urihttp://hdl.handle.net/10722/294070-
dc.description.abstractEnsuring the positive definiteness and avoiding ill conditioning of the Hessian update in the stochastic Broyden-Fletcher-Goldfarb-Shanno (BFGS) method are significant in solving nonconvex problems. This article proposes a novel stochastic version of a damped and regularized BFGS method for addressing the above problems. While the proposed regularized strategy helps to prevent the BFGS matrix from being close to singularity, the new damped parameter further ensures the positivity of the product of correction pairs. To alleviate the computational cost of the stochastic limited memory BFGS (LBFGS) updates and to improve its robustness, the curvature information is updated using the averaged iterate at spaced intervals. The effectiveness of the proposed method is evaluated through the logistic regression and Bayesian logistic regression problems in machine learning. Numerical experiments are conducted by using both synthetic data set and several real data sets. The results show that the proposed method generally outperforms the stochastic damped LBFGS (SdLBFGS) method. In particular, for problems with small sample sizes, our method has shown superior performance and is capable of mitigating ill-conditioned problems. Furthermore, our method is more robust to the variations of the batch size and memory size than the SdLBFGS method.-
dc.languageeng-
dc.publisherInstitute of Electrical and Electronics Engineers. The Journal's web site is located at http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=72-
dc.relation.ispartofIEEE Transactions on Neural Networks and Learning Systems-
dc.rightsIEEE Transactions on Neural Networks and Learning Systems. Copyright © Institute of Electrical and Electronics Engineers.-
dc.rights©20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.-
dc.subjectDamped parameter-
dc.subjectlimited memory BFGS (LBFGS)-
dc.subjectnonconjugate exponential models-
dc.subjectnonconvex optimization-
dc.subjectstochastic quasi-Newton (SQN) method-
dc.titleA Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With Applications-
dc.typeArticle-
dc.identifier.emailWu, HC: hcwueee@hku.hk-
dc.identifier.emailChan, SC: scchan@eee.hku.hk-
dc.identifier.emailLam, WH: whlam@HKUCC-COM.hku.hk-
dc.identifier.authorityChan, SC=rp00094-
dc.identifier.authorityLam, WH=rp00136-
dc.description.naturelink_to_subscribed_fulltext-
dc.identifier.doi10.1109/TNNLS.2019.2957843-
dc.identifier.pmid31902778-
dc.identifier.scopuseid_2-s2.0-85086036513-
dc.identifier.hkuros319280-
dc.identifier.volume31-
dc.identifier.issue11-
dc.identifier.spage4776-
dc.identifier.epage4790-
dc.identifier.isiWOS:000587699700029-
dc.publisher.placeUnited States-

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats