{"id":1165,"date":"2022-01-18T18:54:19","date_gmt":"2022-01-18T18:54:19","guid":{"rendered":"http:\/\/www.wellformedness.com\/blog\/?p=1165"},"modified":"2022-01-21T20:16:03","modified_gmt":"2022-01-21T20:16:03","slug":"logistic-regression-as-the-bare-minimum-or-against-naive-bayes","status":"publish","type":"post","link":"https:\/\/www.wellformedness.com\/blog\/logistic-regression-as-the-bare-minimum-or-against-naive-bayes\/","title":{"rendered":"Logistic regression as the bare minimum. Or, Against na\u00efve Bayes"},"content":{"rendered":"<p>When I teach introductory machine learning, I begin with (categorical) n<em>a\u00efve Bayes<\/em> classifiers. These are arguably the simplest possible supervised machine learning model, and can be explained quickly to anyone who understands probability and the method of maximum likelihood estimation. I then pivot and introduce <em>logistic regression\u00a0<\/em>and its various forms. Ng et al. (2002) provide a nice discussion of how the two relate, and I encourage students to read their study.<\/p>\n<p>Logistic regression is a more powerful technique than na\u00efve Bayes. First, it is &#8220;easier&#8221; in some sense (Breiman 2001) to estimate the conditional distribution, as one does in logistic regression, than to model the joint distribution, as one does in na\u00efve Bayes. Secondly, logistic regression can be learned using standard (online) stochastic gradient descent methods. Finally, it naturally supports conventional regularization strategies needed to avoid overfitting. For this reason, in 2022, I consider regularized logistic regression the bare minimum supervised learning method, the least sophisticated method that is possibly good enough. The pedagogical-instructional problem I then face is trying to convince students <em>not<\/em> to use na\u00efve Bayes, given that it is obsolete\u2014it is virtually always inferior to regularized logistic regression\u2014given that tools like <a href=\"https:\/\/scikit-learn.org\/stable\/\">scikit-learn<\/a> (Pedregosa et al. 2011) make it almost trivial to swap one machine learning method for the other.<\/p>\n<h1><strong>References<\/strong><\/h1>\n<p>Breiman, Leo. 2001. Statistical modeling: the two cultures. <em>Statistical Science<\/em> 16:199-231.<br \/>\nNg, Andrew Y., and Michael I. Jordan. 2002. On discriminative vs. generative classifiers: a comparison of logistic regression and naive Bayes. In <em>Proceedings of NeurIPS<\/em>, pages 841-848.<br \/>\nPedregosa, Fabian, Ga\u00ebl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, &#8230;, and \u00c9douard Duchesnay. 2011. Scikit-learn: machine learning in Python. <em>Journal of Machine Learning Research<\/em> 12:2825-2830.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>When I teach introductory machine learning, I begin with (categorical) na\u00efve Bayes classifiers. These are arguably the simplest possible supervised machine learning model, and can be explained quickly to anyone who understands probability and the method of maximum likelihood estimation. I then pivot and introduce logistic regression\u00a0and its various forms. Ng et al. (2002) provide &hellip; <a href=\"https:\/\/www.wellformedness.com\/blog\/logistic-regression-as-the-bare-minimum-or-against-naive-bayes\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Logistic regression as the bare minimum. Or, Against na\u00efve Bayes&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","footnotes":""},"categories":[3,4,5],"tags":[],"class_list":["post-1165","post","type-post","status-publish","format-standard","hentry","category-dev","category-language","category-nlp"],"_links":{"self":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/1165","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/comments?post=1165"}],"version-history":[{"count":5,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/1165\/revisions"}],"predecessor-version":[{"id":1194,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/1165\/revisions\/1194"}],"wp:attachment":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/media?parent=1165"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/categories?post=1165"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/tags?post=1165"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}