Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-3214
Title: Scalable architecture for big data financial analytics : user-defined functions vs. SQL
Authors : Stockinger, Kurt
Bundi, Nils Andri
Heitz, Jonas
Breymann, Wolfgang
Published in : Journal of Big Data
Volume(Issue) : 6
Issue : 46
Publisher / Ed. Institution : Springer
Issue Date: 2019
License (according to publishing contract) : CC BY 4.0: Attribution 4.0 International
Type of review: Peer review (Publication)
Language : English
Subjects : Financial analytics; Query processing; Performance evaluation; User-defined function
Subject (DDC) : 005: Computer programming, programs and data
332: Financial economics
Abstract: Large financial organizations have hundreds of millions of financial contracts on their balance sheets. Moreover, highly volatile financial markets and heterogeneous data sets within and across banks world-wide make near real-time financial analytics very challenging and their handling thus requires cutting edge financial algorithms. However, due to a lack of data modeling standards, current financial risk algorithms are typically inconsistent and non-scalable. In this paper, we present a novel implementation of a real-world use case for performing large-scale financial analytics leveraging Big Data technology. We first provide detailed background information on the financial underpinnings of our framework along with the major financial calculations. Afterwards we analyze the performance of different parallel implementations in Apache Spark based on existing computation kernels that apply the ACTUS data and algorithmic standard for financial contract modeling. The major contribution is a detailed discussion of the design trade-offs between applying user-defined functions on existing computation kernels vs. partially re-writing the kernel in SQL and thus taking advantage of the underlying SQL query optimizer. Our performance evaluation demonstrates almost linear scalability for the best design choice.
Departement: School of Engineering
Organisational Unit: Institute of Applied Information Technology (InIT)
Publication type: Article in scientific Journal
DOI : 10.1186/s40537-019-0209-0
10.21256/zhaw-3214
ISSN: 2196-1115
URI: https://digitalcollection.zhaw.ch/handle/11475/17286
Published as part of the ZHAW project : Large Scale Data-Driven Financial Risk Modelling
Appears in Collections:Publikationen School of Engineering

Files in This Item:
File Description SizeFormat 
BigDataFinancialAnalyticsUDFvsSQL_JournalBigData.pdf2.16 MBAdobe PDFThumbnail
View/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.