1. komponenta

Lecture type	Total
Lectures	15
Exercises	30

COURSE AIMS AND OBJECTIVES:
- Working with tools for developing complex data pipelines
- Developing aggregation frameworks considering the level of abstraction towards the system user
- Understanding the theory of data flows and implementing real-time data processing systems
- Configuring relational and non-relational databases for scalable data storage and retrieval
- Object-oriented implementation of advanced components for searching relational and non-relational databases
- Designing and implementing data solutions in the cloud

COURSE DESCRIPTION AND SYLLABUS:
1. Development of data pipelines and aggregation frameworks. Docker. Transformations. Types of data pipeline systems. State machine in data pipelines. ETL and ELT processes. Workflow monitoring. Levels of system utilization for processing. Content of auxiliary packages. Levels of data processing parallelization. (4 weeks)
2. Storage of analytical results. Types of aggregations and analysis results. Optimization for storing large amounts of data. Data search systems. Interfaces for generating images and reports. (3 weeks)
3. Stream data processing. Motivation and basics of data streams. Message systems. Window operations. Triggers and watermarks. Stream data processing systems. Kappa, Lambda, and hybrid architectures. (3 weeks)
4. Cloud data engineering. Basics of cloud engineering. Cloud infrastructure. Cloud security. Implementation of data processing systems in the cloud. (3 weeks)

1. Streaming systems: the what, where, when, and how of large-scale data processing, http://www.streamingbook.net, Akidau, T., Chernyak, S., & Lax, R., O'Reilly Media, Inc., 2018.
2. Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems, https://dataintensive.net/, Kleppmann, M., O'Reilly Media, Inc., 2017.
3. Big Data, Cloud Computing, and Data Science Engineering (Vol. 844), https://www.barnesandnoble.com/w/big-data-cloud-computing-and-data-science-engineering-roger-lee/1133107543, Lee, R. (Ed.), Springer, 2019.
4. CS 591 K1: Data Stream Processing and Analytics, https://vasia.github.io/dspa21/index.html, Boston University, 2021.
5. CSE2520: Big Data Processing, https://burcuku.github.io/cse2520-bigdata/, TU Delft, 2022.

Enrollment :
Passed : Database systems

Izborni predmet 4, 5, 6 - Regular study - Financial and Business Mathematics

Izborni predmet 4, 5, 6 - Regular study - Financial and Business Mathematics