title: Textual trends detection at OK
metadata.name: dc.contributor.author
metadata.value: Malyutin, Evgeniy A.
metadata.lang: -
metadata.name: dc.contributor.author
metadata.value: Bugaichenko, Dmitriy Yu.
metadata.lang: -
metadata.name: dc.contributor.author
metadata.value: Mishenin, Alexey N.
metadata.lang: -
metadata.name: dc.date.accessioned
metadata.value: 2017-10-12T09:57:20Z
metadata.lang: -
metadata.name: dc.date.available
metadata.value: 2017-10-12T09:57:20Z
metadata.lang: -
metadata.name: dc.date.issued
metadata.value: 2017-09
metadata.lang: -
metadata.name: dc.identifier.citation
metadata.value: Malyutin E. A., Bugaichenko D. Y., Mishenin A. N. Textual trends detection at OK. Vestnik of Saint Petersburg University. Applied Mathematics. Computer Science. Control Processes, 2017, vol. 13, iss. 3, pp. 313–325.
metadata.lang: en_GB
metadata.name: dc.identifier.other
metadata.value: 10.21638/11701/spbu10.2017.308
metadata.lang: -
metadata.name: dc.identifier.uri
metadata.value: http://hdl.handle.net/11701/8516
metadata.lang: -
metadata.name: dc.description.abstract
metadata.value: Social networks now serve not as a mere medium for entertainment, but as an information
  distribution channel that is replacing classical mass media. In this article we describe a
  scalable trend detection system implemented with the social network OK. Actors (users and
  communities) of social networks form a broad agenda. The content of social networks is specific:
  • UGC (user generated content) is difficult to process;
  • actors generate a multilingual text. This requires attracting a large number of highly paid
  professionals in the case of classical media analysis;
  • modern social networks comprise a highly-connected society with high “response time”.
  Therefore, the system must work in real time;
  • social networks are used by spammers as a platform for promotion and obtrusive
  advertising, therefore the system should contain the ability to filter spam content.
  Applying standard methods of media analysis to this seems impossible. It creates a natural
  demand for developing and implementing textual trend detection and analysis software. There
  are two main approaches of trend detection in academic papers: topic modeling (and further
  topics evolutionary analysis) and distributive models based on frequency-like properties of
  distinct terms. We conducted an analysis of scientific papers using both approaches taking
  into account the specific features of social networks. As a result of research, it was decided to
  use distributive models as a base for the system development. OK is one of the largest social
  networks in Russia and the CIS countries. Actors generate over 100M symbols of text every
  day. Even basic processing is a serious technical problem. So we are forced to use Big Data
  approaches through the development. We introduce lambda-architecture based on three main
  components:
  • daily-batch processing component, based on Apache Spark;
  • streaming processing component, based on Apache Samza;
  • mini-batch processing component, based on Spark Streaming.
  The article describes in detail the architecture and technical features of each component. In
  conclusion we present the results of operating the system as well as discuss areas for further
  research and development. Refs 13. Figs 7. Table 1.
metadata.lang: en_GB
metadata.name: dc.language.iso
metadata.value: ru
metadata.lang: en_GB
metadata.name: dc.publisher
metadata.value: St Petersburg State University
metadata.lang: en_GB
metadata.name: dc.relation.ispartofseries
metadata.value: Vestnik of St Petersburg University. Applied Mathematics. Computer Science. Control Processes;Volume 13; Issue 3
metadata.lang: -
metadata.name: dc.subject
metadata.value: natural language processing
metadata.lang: en_GB
metadata.name: dc.subject
metadata.value: trend detection
metadata.lang: en_GB
metadata.name: dc.subject
metadata.value: big data
metadata.lang: en_GB
metadata.name: dc.title
metadata.value: Textual trends detection at OK
metadata.lang: en_GB
metadata.name: dc.type
metadata.value: Article
metadata.lang: en_GB
resources.filename: 08-Malyutin.pdf
resources.url: https://dspace.spbu.ru/bitstream/11701/8516/1/08-Malyutin.pdf
resources.format: Adobe PDF
resources.size: 713,62 kB