Bussiness Intelligence Orientada als Serveis

Esteu aquí

Crèdits
6
Tipus
Complementària d'especialitat (Enginyeria de Serveis)
Requisits
Aquesta assignatura no té requisits, però té capacitats prèvies
Departament
ESSI
The main goal of this course is to analyze the technological and engineering needs of Big Data Management. The enabling technology for such a challenge is cloud services, which provide the elasticity needed to properly scale the infrastructure as the needs of the company grow. Thus, students will learn advanced data management techniques (i.e., NOSQL solutions) that also scale with the infrastructure. Being Big Data Management the evolution of Data Warehousing, such knowledge (see the corresponding subject in Data Science speciality for more details on its contents) is assumed in this course , which will specifically focus on the management of data Volume and Velocity.

On the one hand, to deal with high volumes of data, we will see how a distributed file system can scale to as many machines as necessary. Then, we will study different physical structures we can use to store our data in it. Such structures can be in the form of a file format at the operating system level, or at a higher level of abstraction. In the latter case, they take the form of either sets of key-value pairs, collections of semi-structured documents or column-wise stored tables. We will see that, independently of the kind of storage we choose, current highly parallelizable processing systems using funtional programming principles (typically based on Map and Reduce functions), whose processing framework can rely on temporal files (like Hadoop MapReduce) or in-memory structures (like Spark).

On the other hand, to deal with high velocity of data, we need some low latency system which processes either streams or micro-batches. However, nowadays, data production is already beyond processing technologies capacity. More data is being generated than we can store or even process on the fly. Thus, we will recognize the need of (a) some techniques to select subsets of data (i.e., filter out or sample), (b) summarize them maximizing the valuable information retained, and (c) simplify our algorithms to reduce their computational complexity (i.e., doing one single pass over the data) and provide an approximate answer.

Finally, the complexity of a Big Data project (combining all the necessary tools in a collaborative ecosystem), which typically involves several people with different backgrounds, requires the definition of a high level architecture that abstracts technological difficulties and focuses on functionalities provided and interactions between modules. Therefore, we will also analyse different software architectures for Big Data.

This course participates in a joint project conducted during the second semester together with VBP, SDM and CC. In VBP, the students will come up with a business idea related to Big Data Management and Analytics, which will be evaluated from a business perspective. In the three other courses the students have to implement a prototype meeting the business idea created. In CC, they will be introduced to the main concepts behind large-scale distributed computing based on a service-based model and will have to choose the right infrastructure for their prototype. In BDM and SDM, they will be introduced to specific data management techniques to deal with Volume and Velocity (BDM) and Variety (SDM). As final outcome, a working prototype must be delivered.

Professors

Responsable

  • Alberto Abello Gamazo ( )

Altres

  • Sergi Nadal Francesch ( )

Hores setmanals

Teoria
2
Problemes
0
Laboratori
1
Aprenentatge dirigit
0
Aprenentatge autònom
5.33

Competències

Competències Tècniques Generals

Genèriques

  • CG2 - Capacitat per a dirigir, planificar i supervisar equips multidisciplinaris.
  • CG5 - Capacitat per aplicar solucions innovadores i realitzar avenços en el coneixement que explotin els nous paradigmes de la Informàtica, particularment en entorns distribuïts.

Competències Transversals

Treball en equip

  • CTR3 - Ser capaç de treballar com a membre d'un equip, ja sigui com a un membre més, ja sigui realitzant tasques de direcció, amb la finalitat de contribuir a desenvolupar projectes d'una manera pragmàtica i amb sentit de la responsabilitat; assumir compromisos tenint en compte els recursos disponibles.

Bàsiques

  • CB7 - Que els estudiants siguin capaços d'integrar coneixements i enfrontar-se a la complexitat de formular judicis a partir d'una informació que, essent incomplerta o limitada, inclogui reflexions sobre les responsabilitats socials i ètiques vinculades a l'aplicació dels seus coneixements i judicis.

Competències Tècniques de cada especialitat

Específiques comunes

  • CEC1 - Capacitat per aplicar el mètode científic en l'estudi i anàlisi de fenòmens i sistemes en qualsevol àmbit de la Informàtica, així com en la concepció, disseny i implantació de solucions informàtiques innovadores i originals.
  • CEC2 - Capacitat per al modelatge matemàtic, càlcul i disseny experimental en centres tecnològics i d'enginyeria d'empresa, particularment en tasques de recerca i innovació en tots els àmbits de la Informàtica.
  • CEC3 - Capacitat per aplicar solucions innovadores i realitzar avanços en el coneixement que explotin els nous paradigmes de la Informàtica, particularment en entorns distribuïts.

Objectius

  1. Comprendre les diferencies i beneficis de la gestió de dades a memòria.
    Competències relacionades: CB7, CEC1, CEC2, CEC3, CG5,
  2. Comprendre el fluxe d'execució una consulta distribuida.
    Competències relacionades: CB7, CEC1, CEC2, CEC3, CG5,
  3. Identificar les dificultat de l'escalabilitat i la paral·lelització.
    Competències relacionades: CB7, CEC1, CEC2, CEC3, CG5,
  4. Dissenyar una base de dades distribuïda utilitzant eines NOSQL.
    Competències relacionades: CG2, CB7, CEC1, CEC2, CEC3, CTR3, CG5,
  5. Produir un programa funcional per a processar Big Data en un entorn Cloud.
    Competències relacionades: CG2, CB7, CEC1, CEC2, CEC3, CTR3, CG5,
  6. Gestionar i processar un Stream de dades.
    Competències relacionades: CG2, CB7, CEC1, CEC2, CEC3, CTR3, CG5,
  7. Dissenyar l'arquitectura per a un sistema de gestió de Big Data.
    Competències relacionades: CG2, CB7, CEC1, CEC2, CEC3, CTR3, CG5,

Continguts

  1. Introducció
    Big Data, Cloud Computing, Scalability
  2. Disseny de Big Data
    Polyglot systems; Schemaless databases; Key-value stores; Wide-column stores; Document-stores
  3. Gestió de dades distribuides
    Transparency layers; Distributed file systems; File formats; Fragmentation; Replication and synchronization; Sharding; Consistent hash; LSM-Trees
  4. Gestió de dades a memòria
    NUMA architectures; Columnar storage; Late reconstruction; Light-weight compression
  5. Processament de dades distribuït
    Distributed Query Processing; Sequential access; Pipelining; Parallelism; Synchronization barriers; Multitenancy; MapReduce; Resilient Distributed Datasets; Spark
  6. Gestió i processament de Streams
    One-pass algorithms; Sliding window; Stream to relation operations; Micro-batching; Sampling; Filtering; Sketching
  7. Arquitectures de Big Data
    Centralized and Distributed functional architectures of relational systems; Data Wareshousing architectures; Service Oriented Architecture; Lambda architecture

Activitats

Activitat Acte avaluatiu


Classes de teoria

In these activities, the lecturer will introduce the main theoretical concepts of the subject. Besides lecturing, cooperative learning techniques will be used. These demand the active participation of the students, and consequently will be evaluated.
Objectius: 1 2 3 4 5 6 7
Continguts:
Teoria
28h
Problemes
0h
Laboratori
0h
Aprenentatge dirigit
0h
Aprenentatge autònom
28h

Projecte

The students will be asked to define a BI project either using services or making decisions on a service, and prepare a demo of it in front of the classmates and a jury. Also, there will be some deliverables along the course.
Objectius: 2 3 4 5 6 7
Continguts:
Teoria
0h
Problemes
0h
Laboratori
9h
Aprenentatge dirigit
0h
Aprenentatge autònom
57h

Teoria
2h
Problemes
0h
Laboratori
0h
Aprenentatge dirigit
0h
Aprenentatge autònom
8h

Laboratori

Students will use different NOSQL tools in a sandbox environment.
Objectius: 2 3 4 5 6
Continguts:
Teoria
0h
Problemes
0h
Laboratori
6h
Aprenentatge dirigit
0h
Aprenentatge autònom
12h

Metodologia docent

The course comprises theory, problems, lab sessions and a project.

Theory: The theory classes comprise the teacher's explanations and constitute the main part of the course. The students will also have some contents to be read and prepared outside the classroom, and will be asked to participate in cooperative learning activities to solve some problems.

Lab: There will be some lab sessions to introduce some of the technologies and a cluster execution environment.

Project: The students will have to implement a proof of concept of an analytical service, which will share the topic with the project in other sibling subjects: BIP, SEAIT, VBP, SDM, CC, and WS (for those students taking them).

Mètode d'avaluació

Final mark = 35% C + 30% Pr + 5%P + 30%E

C = Weighted average of collaborative activities and labs
Pr = Project
P = Peer evaluation
E = Exam

Calculation of C:
1) Multiply the mark of each activity/lab by a weight equal to 1, 2, 4 or 6 (depending on the content of the corresponding activity/lab)
2) Divide the sum of these values by the sum of weights assigned minus 4

Calculation of P: Students will have multiple mates in the activities during the semester and they will evaluate them at the end. Based on these evaluations, the teacher will assign a mark to each student.

Project: Students will implement a proof of concept of the project and will present it in front of their classmates and a jury. Based on that presentation, and the delivered materials during the course the teacher will assign a mark to each student.

Bibliografia

Bàsica:

Complementaria:

Capacitats prèvies

Being Big Data Management the evolution of Data Warehousing, such knowledge is assumed in this course. Thus, general knowledge is expected on: Relational database desing; Database management system architecture; ETL and OLAP

Specifically, knowledge is expected on:
- Multidimensional modeling (i.e, star schemas)
- Querying relational databases
- Physical design of relational tables (i.e., partitioning)
- Hash and B-tree indexing
- External sorting algorithms (i.e., merge-sort)
- ACID transactions