{"id":898,"date":"2020-03-27T17:16:47","date_gmt":"2020-03-27T17:16:47","guid":{"rendered":"http:\/\/www.wellformedness.com\/blog\/?p=898"},"modified":"2020-03-27T17:23:06","modified_gmt":"2020-03-27T17:23:06","slug":"pynini-2020-state-of-the-sandwich","status":"publish","type":"post","link":"https:\/\/www.wellformedness.com\/blog\/pynini-2020-state-of-the-sandwich\/","title":{"rendered":"Pynini 2020: State of the Sandwich"},"content":{"rendered":"<p>I have been meaning to describe some of the work I have been doing on <a href=\"http:\/\/www.opengrm.org\/twiki\/bin\/view\/GRM\/Pynini\">Pynini<\/a>, our weighted finite-state grammar development platform. For one, while I have been the primary contributor through the history of the project (Richard Sproat wrote the excellent<a href=\"http:\/\/www.opengrm.org\/twiki\/bin\/view\/GRM\/PyniniPathsDoc\"> path iteration library<\/a>), we are now also getting many contributions from Lawrence Wolf-Sonkin (rewrite of the symbol table wrapper, type hints) and lots of usability and bug reports from the Google linguists.<\/p>\n<p>We are currently on Pynini release 2.1.1. Here are some new features\/improvements from the last few releases:<\/p>\n<ul>\n<li>2.0.9: Adds an efficient multi-argument <code>union<\/code>.<\/li>\n<li>2.0.9: Pynini (and the rest of <a href=\"http:\/\/www.opengrm.org\/twiki\/bin\/view\/GRM\/PyniniPathsDoc\">OpenGrm<\/a>) are available on <a href=\"https:\/\/docs.conda.io\/en\/latest\/\">Conda<\/a> via <a href=\"https:\/\/conda-forge.org\/\">Conda-Forge<\/a>. This means that for most users, there is no longer any need to compile Pynini by hand; instead Pynini is compiled (for a variety of platforms) in the cloud, using a continuous integration framework.<\/li>\n<li>2.1.0: Rewrites the string compiler so that symbol tables are no longer attached to compiled FSTs, eliminating the need for expensive symbol table merging and relabeling options.<\/li>\n<li>2.1.0: Rewrites the FST and symbol table class hierarchies to better reflect the organization of lower-level APIs.<\/li>\n<li>2.1.1: Adds <a href=\"https:\/\/www.python.org\/dev\/peps\/pep-0484\/\">PEP 484<\/a>\/<a href=\"https:\/\/www.python.org\/dev\/peps\/pep-0561\/\">PEP 561<\/a>-compatible type stubs.<\/li>\n<\/ul>\n<p>We also have removed or renamed quite a few features:<\/p>\n<ul>\n<li><code>stringify<\/code> is renamed <code>string<\/code>.<\/li>\n<li><code>text<\/code> is renamed <code>print<\/code>\u00a0(cf. the command-line tool <code>fstprint<\/code>).<\/li>\n<li>The <code>defaults<\/code> struct is removed, though it may be reintroduced as a context manager at some point.<\/li>\n<li>The <code>*<\/code> infix operator, previously used for composition is removed; use <code>@<\/code> instead.<\/li>\n<li><code>transducer<\/code>&#8216;s arguments <code>input_token_type<\/code> and <code>output_token_type<\/code> are merged as <code>token_type<\/code>.<\/li>\n<\/ul>\n<p>Finally, we have broken Python 2.7 compatibility as of 2.1.0; <code>pywrapfst<\/code>, the lower-level API, still has some degree of Python 2.7 compatibility, but this is probably the last release to maintain that property.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have been meaning to describe some of the work I have been doing on Pynini, our weighted finite-state grammar development platform. For one, while I have been the primary contributor through the history of the project (Richard Sproat wrote the excellent path iteration library), we are now also getting many contributions from Lawrence Wolf-Sonkin &hellip; <a href=\"https:\/\/www.wellformedness.com\/blog\/pynini-2020-state-of-the-sandwich\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Pynini 2020: State of the Sandwich&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","footnotes":""},"categories":[3,5,8],"tags":[],"class_list":["post-898","post","type-post","status-publish","format-standard","hentry","category-dev","category-nlp","category-python"],"_links":{"self":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/898","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/comments?post=898"}],"version-history":[{"count":4,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/898\/revisions"}],"predecessor-version":[{"id":902,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/898\/revisions\/902"}],"wp:attachment":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/media?parent=898"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/categories?post=898"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/tags?post=898"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}