Social Media Mining on Taipei's Mass Rapid Transit Station Services

based on Visual-Semantic Deep Learning

Department of Transportation Management,

Tamkang University,

No.151, Yingzhuan Rd., Tamsui Dist., New Taipei City 25137

semantic fusion deep learning methods are applied to assess Taipei’s Mass Rapid Transit (MRT) station

services from the internet opinions. An opinion monitoring system includes: (1) opinion mining to build a

social media comment dataset on the ontology of MRT stations.; (2) proposing intent-sentiment, image-text

relationship, and content type categories to assist accessing of passengers’ quality of experience; (3)

constructing a classification model to classify the nature of opinions (4) proposing visualization to provide an

intuitive information display dashboard to help Taipei’s MRT operator sense the sentiment-intention trends of

comments on each station and access the current service level as well as part of the quality management

assessment is also proposed.

Key-Words:- social media analytics, opinion mining, visual semantic, deep learning, Taipei MRT station

Intelligence (AI) becomes a trending topic to public

transportation operators. Operators aim to improve

their service quality and increase the number of

rides. One of the common tactics is to understand

the travelers’ satisfaction, such as by questionnaire,

dealing with customers’ complaint, etc.

Meanwhile, social media now plays an important

role in expressing opinions. According to the 2020

Taiwan Internet Report[1], people are using the

internet during nighttime (18:00~23:59) for Instant

opinion online than through the official platform,

such as leaving a comment on social media, some

may even attach photos alongside. The rise of social

media platforms has provided channels to the

passenger to express their view towards the

transport facilities or other topics they are

concerning about.

When user-generated online comments accumulate

day by day, valuable insights can be discovered by

using social media analytics. During the pandemic

collected about Taipei’s Mass Rapid Transit (MRT)

stations in order to assess MRT station services that

include establishing an opinion classification model

through visual semantic fusion deep learning

methods. An opinion monitoring system that

includes: (1) opinion mining to build a social media

comment dataset on the ontology of railway

transportation stations.; (2) proposing intent-

sentiment, image-text relationship, and content type

categories to assist accessing of travelers’ quality of

level as well as part of the quality management

assessment is also proposed.

2 Literature Review

Social media offers channels to update users status

Social media provide a platform for user-generated

content (UGC). The motivations for people using

and functional purpose. In ref.[6], after investigating

the properties and meanings of tweets, Twitter users

can be divided as information sources, friends,

information seekers, and Twitter users have the need

for daily chatter, conversation, sharing, reporting. In

ref.[7], function and sociality attributes can be found

on ZoneTag, an online photo-sharing social media

polarity classification, opinion target identification,

opinion source identification, opinion

summarization[8].

These UGCs have found to be very useful for

opinion mining. In ref.[9], the study collected

Google Map review on several airports and

summarize 25 latent topics matching the assessment

of Airport Service Quality (ASQ). Compared to

alternative to service quality survey for airports and

online comments towards news article about

comparison of healthcare systems across eight

countries and have found some popular topics

related to healthcare services mentioned and

purposed a national healthcare systems ranking

based on sentiment level.

Over past years, the use of smartphone encourages

more online multimodal opinions that people

comment not only by text but also by attaching

images. The functions of image to the posting text

can be summarized in 11 types of properties[12].

The logico-semantic relation can be classified by the

consists of multiple layers to extract

features(representation) to perform automatic

classification, object recognition and many other

domains[15]. Some famous model includes

recurrent neural network that process well on

sequential text material [16] and convolutional

neural network that process well on images

media[17].

Meanwhile, visual-semantic embedding (VSE) is

the essential technique to input textual and visual

This paper proposes an opinion monitoring system

to assess Taipei’s MRT station services. This

includes opinion mining from the web, data

preprocessing, classification model training, and

finally comment trend visualization interface.

This paper aims to collect passengers’ opinions

toward Taipei’s MRT stations they have been to.

Selecting a suitable station-based social media, the

Taoyuan MRT (those in Taipei) and Tamsui LRT

stations, a total of 135 stations. After removing

meaningless comments and translating comments of

foreign language into Chinese, we have collected

2179 opinion with image and text, ranging from

2017 to 2020.

opinion dataset. This paper classifies each review

into three categories: intent-sentiment, image-text

labels are “Image-text Related” and “Image-text

Unrelated”. This classification has been discussed in

several visual semantic studies to understand the

the main focus of opinion.

3.2 Classification Model to Sort Out

Opinions Automatically

This neural network model identifies the intent-

sentiment, image-text relationship, and content type

of the online opinion towards MRT station services

using a visual semantic deep-learning method.

Rather than manual analysis of opinion, this model

aims at providing a cost-efficient way to sort out

online opinions.

Empirical studies are explained as followings:

processing,

The text on each opinion needs to be converted into

word vector(features) beforehand. Due to the unique

structure of Chinese wordings, text undergoes

segmentation by CKIPtagger, a popular

segmentation tool, to separate sentences into

meaningful wordings. Since most opinions are

focused on MRT stations, a transport-word

supplement dictionary is fed into CKIPtagger for

better segmentation, adding MRT station name and

referred as stopwords consisting mostly of

prepositional conjunctions. The text output is finally

up to extract word vector.

This paper uses Word2Vec Skip-gram modal to

extract text feature because the Skip-gram modal

has better training result on rare words [23], making

it suitable to this transport-terms filled dataset. The

result of text segmentation is input into Word2Vec

Skip-gram model, whose training dimension is set

on 300, bring out the word vector needed for the

channel}, bring out the image representation needed

for the computational model.

3.2.2 Model Build Up

This paper builds up a visual-semantic neural

network by using Python, Tensorflow and Keras as

shown in Fig.2. First, the pre-processed image and

text are input to the visual input layer and the textual

input layer respectively to do encoding, then the

and content type.

Fig. 2: The architecture of the model

previous steps into the current steps. GRU consists

of update gate and reset gate, holding the existing

features while adding new content into current steps.

[24] The GRU-RNN network is used to extract the

features of comments’ word embedding and then

pass to dense layer of dimension 100.

Table 2. Structure of the pre-trained RESNET V2

Model

The visual input layer takes the preprocessed image

representation and undergoes through pre-trained

ResNet101 V2 model[17], an advanced

convolutional neural network(CNN) that consists of

convolution layers, pooling layers, Batch

Normalization, activation function layers, full

connected layers as well as shortcut connections and

into a common multimodal embedding space of

dimensions 212. The fused embedding is further

converged in the fully connected layer of

dimensions 212, 128, and 64. At last, the output

layer consists of three different layers that give out

the comments' intent-sentiment, image-text

3.2.3 Training

80% of the data(1743 samples) is split into training

set, and 20% of data is split into verification set

(436 samples). Since the data is an imbalanced

in the total dataset, therefore the data is split while

performed split to final evaluation. We trained with

epochs and batch size of 200 and 7, respectively.

For evaluation, we reported the classification

accuracy and also F1-score as shown in Table 3.

Both the accuracy and F1-score indicators show that

the visual-semantic model has a predictive ability.

The predictive effect is in an order of Image-Text

Relation (72.7%), Intent-Sentiment (73.9%),

Content Type (61.2%). The performance is similar

to that of some multimodal model like intent

classifying [19].

Class-wise performances are shown as Tables 4-6.

The performance of intent-sentiment classification

reflects that the model can classify some

characteristics of different intent, which the

Integrating with the opinion mining classification

model, it can be utilized to overview any emergent

comment and display the results to the MRT

distribution of intent-sentiment, content type,

image-text relationship by selecting single or

multiple station(s) on the map. The results are then

shown on the right of the interface.

Take the MRT stations in the Tamsui District, the

terminus section of the MRT Red Line, as an

example. The intent-sentiment in this area is shown

in Fig. 4. The majority were neutral comments,

followed by positive comments. This matches the

(descriptive) comments that mainly depict the public

art installations in the Hongshulin station.

4 Conclusion

This paper gathered online comments from the

Google Map of each tagged MRT station in the

Taipei metropolitan area and demonstrated how

these online sources fulfill the characteristics of

This paper classifies each comment into three

text Related” and “Image-text Unrelated” labels.

"Content Type" can be divided into “Station-

related”, “Scenery” and “Local” labels.

Tamsui District area from 2017 to 2020

To monitor the comments of each MRT station, this

paper has proposed a classification to sort out those

comments from the scope of quality management,

and built up a visual-semantic deep learning

approach to foster the process.

The empirical study proved a satisfying predictive

by the other two categories.

under-score stations.

on the Google map. This help analyzes the opinion

mining on the ontology of railway transportation

station since the review come from particular place

tag that greatly reduce the process to identify the

place subject.

This paper has also contributed in demonstrating the

use of visual semantic deep learning model. The

unsorted reviews contain both image and review

text. The model establishes the visual and textual

input layer, fusion layer that concatenate the

difficulty of accessing passenger attitude toward the

metro service. Since this paper proposes an online

opinion mining, this is a contactless passenger

investigation and more cost-friendly solution to

perform a service quality survey under these

circumstance. Metro operators are able to monitor

the status of each station from the passengers’

reviews and look for possible causes of stations with

low number of positive rating, with visualization

tools.

4.2 Limitations and Future Study

review has many neutral comments resulting of a

skewed opinion dataset. This may restrict the scope

of the passengers’ review received. For future

studies, the data sources should expand to several

social media platforms. There are many platforms

with place tag, such as Facebook, Twitter, Instagram.

It would be valuable to collect more comprehensive

information.

The performance of the classification model can be

improved. The accuracy is skewed as mentioned,

another important railway system for long-haul

commuters, can be included in the assessment of the

city-wide railway transportation system. This may

bring more interesting results of the sentiment level

between different type of railway stations.

For future studies, the interactions between online

comments and the real-world situation can also be

further discussed, such as how the negative

comments on specific service attributes (e.g.

temperature, tidiness) can correlate to the station

[2] National Development Council, “2018

Individual/Household Digital Opportunity

Survey in Taiwan”, 2018

[3] B. Hogan. “The Presentation of Self in the

Age of Social Media: Distinguishing

Performances and Exhibitions Online.”

Bulletin of Science, Technology & Society

30: 377 – 386, 2010.

[4] M. Lucie and S. Josef, “Goffman's Theory as

a Framework for Analysis of Self Presentation

82(4):968-982, 2005

[6] A. Java, X. Song, T. Finin, and B. Tseng,

“Why we twitter,” Proceedings of the 9th

WebKDD and 1st SNA-KDD 2007 workshop

on Web mining and social network analysis -,

2007.

[8] K. Khan, B. Baharudin, A. Ullah, “Mining

opinion components from unstructured

reviews: A review”, Journal of King Saud

University-Computer and Information

Sciences, 26(3), 258-275,2014

[9] K. Lee and C. Yu, “Assessment of Airport

Service Quality: A complementary approach

to measure perceived service quality based on

content using natural language processing: a

case study of public satisfaction with

healthcare systems.”, Journal of Computer

Social Science, 2021.

[12] E. E. Marsh and M. Domas White, “A

taxonomy of relationships between images

and text,” J. Doc., vol. 59, no. 6, pp. 647–672,

2003.

[13] R. Martinec, “A system for image-text

relations in new (and old) media,” Vis.

and image: Determining multimodal

document intent in Instagram posts,” arXiv

[cs.CV], 2019.

https://hdl.handle.net/11296/9p2xrb

[21] Y. Tsai, “Internet Public Opinion Sentiment

Analysis on Topic of Taiwan Freeway’s

Distance-based Toll Collection Using Three-

way Decisions Theory”, 2016. Accessed on

2021. [online]. Available:

https://hdl.handle.net/11296/6k9272

[22] T. H. Park, J. Li, H. Zhao, and M. Chau,

[24] J. Chung, C. Caglar, et. Al. “Empirical

Evaluation of Gated Recurrent Neural

Networks on Sequence Modeling.”, arXiv

[cs.CL], 2014