Name : Swadesh

Roll No: 19BM6JP63

Case: Value Creation Through Big Data Analytics at the Bombay Stock Exchange

Bombay stock Exchange (BSE) is Asia's first trade stock exchange centre. In 2017, the BSE had a
market capitalisation of more than $ 1.7 million with 5500 companies listed. Data at BSE was
growing at fast pace, generating around 350 GB per day. BSE had a proprietary data warehouse
and used SAS for performing analytics on the data set. The cost of running and maintaining with
old data warehouse and SAS system was huge. Because of the exponential growth in data and
trading speed, the data points will increase manifolds. To keep up with it BSE will require to
have 10 or 20 times larger data warehouse because of which there will be huge spike in the
cost too. In the old data warehouse, data existed in silos and was scattered across the
information layer. Thus performing task such as report generation, historical analysis etc. was
time consuming. It was a difficult task to perform queries in the given environment and scaling
up was also troublesome. BSE choose Hadoop Distributed File System (HDFS) architecture for
implementation of big data project. Hadoop is an open source technology in which data can be
stored in different clusters built of relatively cheap commodity hardware. The MapReduce
programming model and HDFS are designed in a way that it can handle downtime efficiently.
Switching to this open source platform resulted in cost reduction and improved performance
level. The costs were reduced by 20 % resulting in saving of 50 million. Reports which were
taking hours to generate were done in minutes. The big data implementation helped in
consolidation of information from multiple sources into scalable and robust data repository. It
helped BSE become a data driven enterprise whereby it can act based on real time data.
But having relied on proprietary techonologies for so long there were some risk associated with
implementation of an open source framework:
 Apprehension from few sections of organisation whether open source could eficiently
handle the process or not.
 The proprietary systems were fault tolerant and people at BSE did not wanted to
experiment with techonology fearing it could result in loss of data.

Choosing the right implemention partner is as much as important as choosing the right
techonolgy. The partnership between DataMetica and BSE was one of the key reason for
successful implementation of this project. The BSE and DataMetica's team gelled really well
with BSE providing business expertise and DataMetica providing technical expertise. There
were lot of issues which could have come in the process of implenting the big data architecture
at BSE but all was taken care of because of dilligent approach of Datametica and BSE IT team.
DataMetica was one of the Top 20 big data global companies with clients including Fortune 500
companies. In past, they have deployed enterprise level data hub environment based on open
source technologies, provided the governance mechanism and offered analytics functionalities.
They had a team comprised of delivery manager, solution architect and data scientists. A
delivery manager manage the entire project plan based on client's need. A solution architect
focused on development and implementation of solution architecture to be implemented and
proposed various technologies to be employed. They initally did a POC with BSE on almost zero
cost invovled. It was a win win situation for both of them. DataMetica were very swift and
innovative in their approach.They focused on giving solutions, understanding the business and
quickly developed the prototype. Basically DataMetica effectively ticked all the boxes which BSE
asked for and was the right partner for the project.

Rumour Detection: Rumour detection was an integral part of BSE day to day business. It
identified news on available in public domains regarding a certain company which could affect
the stock prices and categorised them as rumours if confirmed otherwise by the company. BSE
used it's already in place big data platform to store and process data from various available
resources for news such as online articles, Twitter etc. BSE also had it's historical data which
was classified manually as rumours. They built a pool of keywords whose presence generally
suggested a financial rumour. Using this set and domain knowledge from BSE, DataMetica build
a model based on ML and NLP to classify rumours automatically. . After the implementation of
the model the classification accuracy of 90% was achieved by training and refining it iteratively.
For analysis the data from Twitter, extra measures were taken regarding the acronyms, spelling
mistakes and shortened words while building the keywords set. The notion of hashtags were
also incorporated in the model
A process which was an exhaustive manual task and could only be performed once in a day with
offline materials had now become an automated process. Post implementation of this solution
the process could be carried out several times in a day. This freed employees from their regular
scanning work and helped them in focussing on additional complex analysis resulting in
increase of their productivity.

Road Ahead for Big Data & New technologies collaboration:

 BSE should also leverage other departments which are still running on traditional
databases to move to open source data base to reduce the operating cost. The project
should be implemented by POC and then in phases (same they as their current project).
 BSE can leverage big data and its analytical model to gather data for its rumour
detection model. It can use other social media sites such as facebook for data gathering.
This will act an extra input parameter to the rumour detection model for decision
making.
 RPA (Robotic Process Automation) can be used to automate the whole process starting
from Rumour detection(already is) to verification of news from the concern company
and then hosting that data on the BSE's website
 BSE should start providing oportunities to it's employees to learn the benefits of big
data and new technologies to reduce friction while adopting
 Since the data is stored in the open source, BSE should spend on security and
authorization of the databases