title: Chapter 2. Token attribution
metadata.name: dc.contributor.author
metadata.value: Kuznetsov, Sergey A.
metadata.lang: -
metadata.name: dc.contributor.author
metadata.value: Skrebtsova, Tatiana G.
metadata.lang: -
metadata.name: dc.contributor.author
metadata.value: Suvorov, Sergey G.
metadata.lang: -
metadata.name: dc.contributor.author
metadata.value: Klementyeva, Anna V.
metadata.lang: -
metadata.name: dc.date.accessioned
metadata.value: 2020-12-02T10:25:41Z
metadata.lang: -
metadata.name: dc.date.available
metadata.value: 2020-12-02T10:25:41Z
metadata.lang: -
metadata.name: dc.date.issued
metadata.value: 2019
metadata.lang: -
metadata.name: dc.identifier.other
metadata.value: https://doi.org/10.21638/11701/9785288059278.03
metadata.lang: -
metadata.name: dc.identifier.uri
metadata.value: http://hdl.handle.net/11701/21176
metadata.lang: -
metadata.name: dc.description.abstract
metadata.value: The aim of this step is to check the previously assigned token types and to provide the
  letter tokens with morphological information, i.e. values of the relevant grammatical
  categories. As far as word forms are concerned, the procedure is normally called morphological analysis. However, since a text may contain other token types (e.g. abbreviations, formulae, Internet hyperlinks, phone numbers), it is generally referred to as
  token attribution. In the chapter, a wide range of token types are considered from the
  processing viewpoint. In particular, the letter token analysis presupposes a search in a
  number of dictionaries. Apart from a regular Russian morphological dictionary, search
  is also performed in the dictionaries of abbreviations, of fixed phrases, and of proper
  names (personal, geographical, etc.). Morphological analysis often yields more than a
  single attribution. In some cases, the ambiguity can be reduced by taking into account
  graphematical information, but most often, it will remain and pose further problems for
  the syntactic analysis. If the search for a letter token fails in all the dictionaries, the algorithm tries to identify its lemma and predict the grammatical meaning. The attribution
  of other token types is made by mapping them onto a range of patterns. Typical problems
  of both operations are discussed.
metadata.lang: en_GB
metadata.name: dc.language.iso
metadata.value: ru
metadata.lang: en_GB
metadata.name: dc.publisher
metadata.value: St Petersburg State University
metadata.lang: en_GB
metadata.name: dc.subject
metadata.value: morphological analysis
metadata.lang: en_GB
metadata.name: dc.subject
metadata.value: tokenization
metadata.lang: en_GB
metadata.name: dc.subject
metadata.value: abbreviation
metadata.lang: en_GB
metadata.name: dc.subject
metadata.value: fixed phrase
metadata.lang: en_GB
metadata.name: dc.subject
metadata.value: proper name
metadata.lang: en_GB
metadata.name: dc.subject
metadata.value: morphological prediction
metadata.lang: en_GB
metadata.name: dc.title
metadata.value: Chapter 2. Token attribution
metadata.lang: en_GB
metadata.name: dc.type
metadata.value: Book chapter
metadata.lang: en_GB
resources.filename: 27-43.pdf
resources.url: https://dspace.spbu.ru/bitstream/11701/21176/1/27-43.pdf
resources.format: Adobe PDF
resources.size: 645,79 kB