MTL Toolbox

From Taioaan Wiki
Jump to navigation Jump to search

MTL Toolbox (https://learntaiwanese.org/MTLtoolbox/about.html) is software and data to help people use written Taiwanese in Modern Literal Taiwanese (MLT) and other Latin-script writing systems.

Features

How to perform a full-text search

Steps to do a full-text search on our set of Taiwanese dictionaries:

  • Visit "Taiwanese–English dictionaries full-text search" [1]:
  • Select or deselect which dictionaries to search using the checkboxes. By default, all seven dictionaries are included, as well as "DFT_lk", which is the database table for DFT examples.
  • Input search terms to define your search. Typical inputs include English terms, MLT-style syllables (citation tone, before tone sandhi), and number of syllables. Feel free to try any other terms that would help narrow down your search. For example:
    • You searched for apples, but didn't get many results. Instead, search for apple*, to get results with any strings that start with "apple".
    • If you want only results starting with MLT consonant "ph", include ph*: apple* ph*
  • In some cases, it is better to specify a column for a term, especially if it could match in multiple columns. To specify the column to search against, follow the column-name by a ":" character, then the term. For example:
    • Suppose your search term is "too", which is a valid English word but is also a valid MLT syllable. If you meant the English word, type en:too. If you meant the MLT syllable, type u:too ("u" stands for "unjoined").
    • Suppose you want results that contain the very common MLT syllable "lie", which is also a valid word in English. Again, specify the "u" column: u:lie. However, the number of results is still large. If you want only four-syllable results, add ns:4. The entire search input is then u:lie ns:4
    • See #Technical notes for more details.

How to use the MLT segmenter

We describe how to use "Taiwanese–English dictionaries: MLT segmenter & full-text search" [2]. You might want to use this tool if you want to search the dictionaries using MLT but don't want to figure out the citation-tone syllables yourself. As most Taiwanese words written in MLT will not have a hyphen or other character to delimit syllable boundaries, computing the citation tone syllables is a non-trivial process. The MLT segmenter lowers the barrier between MLT and the dictionaries, which have not been completely converted to MLT.

To get started, enter your MLT word, then press "Go". The segmenter will process your input. If successful, it will also run a search. Results from HTB and DFT are displayed, and results from other dictionaries are summarized and linked to.

Typical usage

  • Input: Taiwanese word (typically disyllable: two syllables joined by tone sandhi)
    • Example: køefcie (if ø is not convenient, use 0 (zero) instead: k0efcie)
  • Press return or tap "Go"
    • your input is "unjoined" (original syllables found by database lookup)
      • in this example, the original syllables are: køea and cie
    • search is performed using the unordered collection of syllables (we refer to this as "bag-of-syllables" (BOS). see bag-of-words model)
    • confirm that the results are basically the same (except for HTB which is not unjoined) as if you had run a full-text search for either: køea cie, or: cie køea

Monosyllable

  • if the syllable is a DFT monosyllable, a navigation bar displays adjacent DFT monosyllables in alphabetical order
  • due to high number of matches, "monosyllable mode" displays monosyllable search results. To see all matching results, click on [[🔍 ]]

Other columns

  • The "en:" button will prepend "en:" to your search, causing the following token to be matched against the English column.
  • We haven't implemented Chinese text segmentation, which is non-trivial (see Chinese word-segmented writing).
  • DFT now supports using individual Harnji as tokens. For example, 對 頭. The h_j column is the Harnji (hj) column with spaces between characters.

Data

Local copies of:

Note: The M-fields of DFT and MK are largely machine-generated ("auto-joined") and do not attempt to indicate prescriptive spellings. In some cases, the common or recommended spelling may be different from what is shown. Often, the difference is about apostrophes and/or hyphens.

We also support searching other websites with conversion to POJ/TL:

Technical notes

Our full-text search is provided by the SQLite FTS4 extension. We currently use the Standard Query Syntax. One of the three basic query types supported by FTS tables is "token or token prefix queries":

  • Specify a token prefix by appending an asterisk ('*') to the prefix. (While similar to wildcard character in operating systems, wildcard search is not currently supported by FTS)
  • Specify a column-name followed by a colon (':')
    • Example: hj:頭* (returns entries where Taiwanese written with Harnji begins with character for thaau)
  • Prefix the token with carat ('^') to require token to be very first token in its column

Tokenizer: the default tokenizer ("simple") is used. Because it only does case folding of ASCII characters, Ø and ø do not match each other. Some words starting with Ø include Ørciw, Ørmngg, and Ørtøexli.

See also

Acknowledgements

Retrieved from "https://taioaan.org/wiki/index.php?title=MTL_Toolbox&oldid=57122"