Streaming Fraud Detection: A Kafka-Spark Machine Learning Approach for Banks

Authors

  • REDOUANE HAKIMI Institut National de Statistique et d'Economie Appliquée image/svg+xml
  • Youssef Tajani Institut National de Statistique et d'Economie Appliquée image/svg+xml
  • Mohamed Ouzineb Institut National de Statistique et d'Economie Appliquée image/svg+xml
  • Karim El Moutaouakil Sidi Mohamed Ben Abdellah University image/svg+xml

Keywords:

Apache Kafka, Apache Spark, Fraud Detection, Machine Learning, Banking Security, Real-Time Processing

Abstract

The growth of digital payments requires fraud detection systems that can process high-velocity transaction streams while keeping the number of false alarms manageable. This research proposes a real-time solution that uses Apache Kafka for data streaming, Apache Spark for distributed processing and Spark MLlib for machine learning (ML). Unlike traditional batch methods, the proposed system combines batch and real-time processing layers in a Lambda architecture, deployed as Docker micro-services. The system is evaluated on confidential card-transaction data from a Moroccan bank with an imbalance ratio of 675:1. Decision Trees (DT), Random Forests (RF) and Gradient-Boosted Trees (GBT) were trained on data rebalanced with a hybrid resampling strategy (random under-sampling and SMOTE) and tuned with the Tree-structured Parzen Estimator (TPE) using G-means as the objective. TPE reduced tuning time by 80\% relative to exhaustive grid search for DT, and made the tuning of RF and GBT feasible where grid search could not be completed. The TPE-tuned Random Forest achieved the best trade-off (precision 0.343, recall 0.781, PR AUC 0.612, MCC 0.516) and about 91\% fewer false alarms than its default configuration. End-to-end latency ranged from 3.5 to 5 seconds. The operational cost of the remaining false positives and the limitations of the study are discussed.

References

R. Hakimi, B. Benyacoub, M. Ouzineb, Unveiling the power of cost sensitivity: A comparative study of different models on imbalanced datasets in machine learning, in: Communication and Information Technologies through the Lens of Innovation, Springer Nature Switzerland, 2025, pp. 113–120. doi:10.1007/978-3-031-74470-9_14.

T. Watthaisong, K. Sunat, N. Muangkote, Comparative evaluation of imbalanced data management techniques for solving classification problems on imbalanced datasets, Statistics, Optimization & Information Computing 12 (2) (2024) 547–570. doi:10.19139/soic-2310-5070-1890.

E. Osegi, E. Jumbo, Comparative analysis of credit card fraud detection in simulated annealing trained artificial neural network and hierarchical temporal memory, Machine Learning with Applications 6 (2021) 100080. doi:10.1016/j.mlwa.2021.100080.

A. Izotova, A. Valiullin, Comparison of poisson process and machine learning algorithms approach for credit card fraud detection, Procedia Computer Science 186 (2021) 721–726. doi:10.1016/j.procs.2021.04.214.

J. Domashova, O. Zabelina, Detection of fraudulent transactions using sas viya machine learning algorithms, Procedia Computer Science 190 (2021) 204–209. doi:10.1016/j.procs.2021.06.025.

G. S. Thejas, S. Dheeshjith, S. Iyengar, N. Sunitha, P. Badrinath, A hybrid and effective learning approach for click fraud detection, Machine Learning with Applications 3 (2021) 100016. doi:10.1016/j.mlwa.2020.100016.

Y. Shen, C. Guo, H. Li, J. Chen, Y. Guo, X. Qiu, Financial feature embedding with knowledge representation learning for financial statement fraud detection, Procedia Computer Science 187 (2021) 420–425. doi:10.1016/j.procs.2021.04.110.

A. Cherif, A. Badhib, H. Ammar, S. Alshehri, M. Kalkatawi, A. Imine, Credit card fraud detection in the era of disruptive technologies: A systematic review, Journal of King Saud University - Computer and Information Sciences 35 (1) (2023) 145–174. doi:10.1016/j.jksuci.2022.11.008.

F. K. Alarfaj, I. Malik, H. U. Khan, N. Almusallam, M. Ramzan, M. Ahmed, Credit card fraud detection using state-of-the-art machine learning and deep learning algorithms, IEEE Access 10 (2022) 39700–39715. doi:10.1109/access.2022.3166891.

A. A. Taha, S. J. Malebary, An intelligent approach to credit card fraud detection using an optimized light gradient boosting machine, IEEE Access 8 (2020) 25579–25587. doi:10.1109/access.2020.2971354.

C. Martín, P. Langendoerfer, P. S. Zarrin, M. Díaz, B. Rubio, Kafka-ml: Connecting the data stream with ml/ai frameworks, Future Generation Computer Systems 126 (2022) 15–33. doi:10.1016/j.future.2021.07.037.

D. R. Torres, C. Martín, B. Rubio, M. Díaz, An open source framework based on kafka-ml for distributed dnn inference over the cloud-to-things continuum, Journal of Systems Architecture 118 (2021) 102214. doi:10.1016/j.sysarc.2021.102214.

E. Nagy, R. Lovas, I. Pintye, Á. Hajnal, P. Kacsuk, Cloud-agnostic architectures for machine learning based on apache spark, Advances in Engineering Software 159 (2021) 103029. doi:10.1016/j.advengsoft.2021.103029.

A. Benítez-Hidalgo, C. Barba-González, J. García-Nieto, P. Gutiérrez-Moncayo, M. Paneque, A. J. Nebro, M. d. M. Roldán-García, J. F. Aldana-Montes, I. Navas-Delgado, Titan: A knowledge-based platform for big data workflow management, Knowledge-Based Systems 232 (2021) 107489. doi:10.1016/j.knosys.2021.107489.

F. Carcillo, A. Dal Pozzolo, Y.-A. Le Borgne, O. Caelen, Y. Mazzer, G. Bontempi, SCARFF : A scalable framework for streaming credit card fraud detection with spark, Information Fusion 41 (2018) 182–194. doi:10.1016/j.inffus.2017.09.005.

R. P. Daksa, A. P. Kemala, A comparative study on real time data streaming for fraud detection using kafka with apache flink and apache spark, Procedia Computer Science 269 (2025) 192–199. doi:10.1016/j.procs.2025.08.272.

M. S. ElSayed, N.-A. Le-Khac, M. A. Albahar, A. Jurcut, A novel hybrid model for intrusion detection systems in sdns based on cnn and a new regularization technique, Journal of Network and Computer Applications 191 (2021) 103160. doi:10.1016/j.jnca.2021.103160.

G. Suchacka, A. Cabri, S. Rovetta, F. Masulli, Efficient on-the-fly web bot detection, Knowledge-Based Systems 223 (2021) 107074. doi:10.1016/j.knosys.2021.107074.

M. de Vos, J. Pouwelse, Contrib: Maintaining fairness in decentralized big tech alternatives by accounting work, Computer Networks 192 (2021) 108081. doi:10.1016/j.comnet.2021.108081.

Y. Luo, X. Zou, D. Liu, W. Peng, X. Wu, Extraction of specific arguments from chinese financial news with out-of-domain samples, Procedia Computer Science 183 (2021) 288–294. doi:10.1016/j.procs.2021.02.061.

D. Malhotra, A. Chug, A modified label propagation algorithm for community detection in attributed networks, International Journal of Information Management Data Insights 1 (2) (2021) 100030. doi:10.1016/j.jjimei.2021.100030.

W. Hilal, S. A. Gadsden, J. Yawney, Financial fraud: A review of anomaly detection techniques and recent advances, Expert Systems with Applications 193 (2022) 116429. doi:10.1016/j.eswa.2021.116429.

A. Dal Pozzolo, G. Boracchi, O. Caelen, C. Alippi, G. Bontempi, Credit card fraud detection: A realistic modeling and a novel learning strategy, IEEE Transactions on Neural Networks and Learning Systems 29 (8) (2018) 3784–3797. doi:10.1109/tnnls.2017.2736643.

H. He, E. Garcia, Learning from imbalanced data, IEEE Transactions on Knowledge and Data Engineering 21 (9) (2009) 1263–1284. doi:10.1109/tkde.2008.239.

K. Niu, Z. Zhang, Y. Liu, R. Li, Resampling ensemble model based on data distribution for imbalanced credit risk evaluation in p2p lending, Information Sciences 536 (2020) 120–134. doi:10.1016/j.ins.2020.05.040.

J. Gama, I. Žliobaite, A. Bifet, M. Pechenizkiy, A. Bouchachia, A survey on concept drift adaptation, ACM Computing Surveys 46 (4) (2014) 1–37. doi:10.1145/2523813.

I. Psychoula, A. Gutmann, P. Mainali, S. H. Lee, P. Dunphy, F. Petitcolas, Explainable machine learning for fraud detection, Computer 54 (10) (2021) 49–59. doi:10.1109/mc.2021.3081249.

S. Lessmann, B. Baesens, H.-V. Seow, L. C. Thomas, Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research, European Journal of Operational Research 247 (1) (2015) 124–136. doi:10.1016/j.ejor.2015.05.030.

I. D. Mienye, N. Jere, Deep learning for credit card fraud detection: A review of algorithms, challenges, and solutions, IEEE Access 12 (2024) 96893–96910. doi:10.1109/access.2024.3426955.

A. Dal Pozzolo, O. Caelen, R. A. Johnson, G. Bontempi, Calibrating probability with undersampling for unbalanced classification, in: 2015 IEEE Symposium Series on Computational Intelligence, IEEE, 2015, pp. 159–166. doi:10.1109/ssci.2015.33.

N. V. Chawla, K. W. Bowyer, L. O. Hall, W. P. Kegelmeyer, Smote: Synthetic minority over-sampling technique, Journal of Artificial Intelligence Research 16 (2002) 321–357. doi:10.1613/jair.953.

A. Fernandez, S. Garcia, F. Herrera, N. V. Chawla, Smote for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary, Journal of Artificial Intelligence Research 61 (2018) 863–905. doi:10.1613/jair.1.11192.

T. Saito, M. Rehmsmeier, The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets, PLOS ONE 10 (3) (2015) e0118432. doi:10.1371/journal.pone.0118432.

D. Chicco, G. Jurman, The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation, BMC Genomics 21 (1) (2020). doi:10.1186/s12864-019-6413-7.

J. Bergstra, Y. Bengio, Random search for hyper-parameter optimization, Journal of Machine Learning Research 13 (2012) 281–305.

J. Bergstra, R. Bardenet, Y. Bengio, B. Kégl, Algorithms for hyper-parameter optimization, in: Advances in Neural Information Processing Systems 24 (NIPS 2011), 2011, pp. 2546–2554.

M. Zaharia, R. S. Xin, P. Wendell, T. Das, M. Armbrust, A. Dave, X. Meng, J. Rosen, S. Venkataraman, M. J. Franklin, A. Ghodsi, J. Gonzalez, S. Shenker, I. Stoica, Apache spark: a unified engine for big data processing, Communications of the ACM 59 (11) (2016) 56–65. doi:10.1145/2934664.

Downloads

Published

2026-11-13

Data Availability Statement

The data are confidential financial transaction records provided by a Moroccan banking institution. Due to non-disclosure agreements and data protection regulations, they cannot be shared.

How to Cite

HAKIMI, R., Tajani, Y., Ouzineb, M., & El Moutaouakil, K. (2026). Streaming Fraud Detection: A Kafka-Spark Machine Learning Approach for Banks. International Journal of Intelligent Optimization and Logistics, 1(1). https://www.ijiol.com/ijiol/article/view/v1i1-a0002