Big Data is traditionally defined with the three V's: Volume, Velocity and Variety. Traditionally, Big Data has been associated with Volume (e.g., the Hadoop ecosystem) and recently Velocity has earned its momentum (especially, with the arrival of Stream processors such as Spark). However,
currently, associating Big Data with simply Volume or Velocity is simply a mistake. The biggest challenge in Big Data Management is nowadays the Variety challenge and how to tackle Variety in real-world projects is yet not clear and there are no standarized solutions (such as Hadoop for Volume or Spark for Velocity) for this challenge.
In this course the student will be introduced to advanced database technologies, modeling techniques and methods for tackling Variety for decision making. We will also explore the difficulties that arise when combining Variety with Volume and / or Velocity. The focus of this course is on the need to enrich the available data (typically owned by the organization) with external repositories (special attention will be paid to Open Data), in order to gain further insights into the organization business domain. There is a vast amount of examples of external data to be considered as relevant in the decision making processes of any company. For example, data coming from social networks such as Facebook or Twitter; data released by governmental bodies (such as town councils or governments); data coming from sensor networks (such as those in the city services within the Smart Cities paradigm); etc.
This is a new hot topic without a clear and established (mature enough) methodology. For this reason, it requires rigorous thinking, innovation and a strong technical background in order to master the inclusion of external data in an organization decision making processes. Accordingly, this course focuses on three main aspects:
1.- Technical aspect. This represents the core discussion in the course and includes:
- dealing with semi-structured or non-structured data (as in the Web),
- the effective use of metadata to understand external data as by means of Linked Data,
- mastering the main formalisms (mostly coming from the Semantic Web) to enrich the data with metadata (ontology languages, RDF, XML, etc.),
- determine relevant sources, apply and use semantic mechanisms to automate the addition (potentially integration), linkage and / or cross of data between heterogeneous data sources,
- refining and visualizing Open Data
2.- Entrepreneurship and innovation, which includes:
- working on the visionary aspect to boost new analytical perspectives on a business domain by considering external sources and,
- developing added value to current systems by means of (such) external data
Person in charge
Oscar Romero Moral (
Jovan Varga (
Generic Technical Competences
CG4 - Capacity for general and technical management of research, development and innovation projects, in companies and technology centers in the field of Informatics Engineering.
CG5 - Capability to apply innovative solutions and make progress in the knowledge to exploit the new paradigms of computing, particularly in distributed environments.
Entrepreneurship and innovation
CTR1 - Capacity for knowing and understanding a business organization and the science that rules its activity, capability to understand the labour rules and the relationships between planning, industrial and commercial strategies, quality and profit. Capacity for developping creativity, entrepreneurship and innovation trend.
CTR3 - Capacity of being able to work as a team member, either as a regular member or performing directive activities, in order to help the development of projects in a pragmatic manner and with sense of responsibility; capability to take into account the available resources.
Technical Competences of each Specialization
CEC1 - Ability to apply scientific methodologies in the study and analysis of phenomena and systems in any field of Information Technology as well as in the conception, design and implementation of innovative and original computing solutions.
CEC3 - Ability to apply innovative solutions and make progress in the knowledge that exploit the new paradigms of Informatics, particularly in distributed environments.
Determine how to apply graph formalisms to solve the Variety challenge (data integration)
Master the main semantic-aware formalisms to enable semantic modeling
Determine how to apply graph formalisms to solve the Variety challenge (data integration)
Reinforce team work capabilities in order to develop innovative solutions by means of complementing the organization data with external data
Perform graph data processing both in centralized and distributed environments
Introduction and formalisation of Variety in Big Data and its management
Definition of data management tasks: from a database perspective and knowledge representation.
Definition of Variety and Big Data. Syntactic and Semantic heterogeneities. Impact of heterogeneities in the identified data management tasks.
Data integration. Theoretical framework for the management and integration of heterogeneous data sources.
Main components of an integration system: data sources, global schema and mappings.
The concept of canonical model for data integration. Definition of data model. Main characteristics of a canonical data model.
Graphs as solution to the Variety challenge
Graphs as the best canonical model for data integration.
Graph data models main features. Differences with other data models (specially the relational data model).
Data and metadata concepts and their formalization in graph models.
Use cases (highlighting topological benefits): fraud detection, bioinformatics, traffic and logistics, social networks, etc.
Introduction to the main graph models: property graph and knowledge graphs.
Property graphs management
Data structures. Integrity constraints.
Basic operations. Based on topology, content and hybrid.
Graph query languages: GraphQL.
Graph database concept: tool heterogeneity when implementing the graph structures. Impact of such decisions in the main operations.
Distributed graph databases. Need and difficulties. Thinking like a vertex paradigm as standard de facto in distributed graph processing.
Main distributed graph algorithms.
Knowledge graph management
Data structure. RDF. Origin and relationship with Linked Open Data. Integrity constraints.
Data structure: RDFS and OWL. Relationship with first order logic. Foundations in Description Logics. Integrity constraints. Reasoning.
Basic operations and query language. SPARQL and underlying algebra. Entailment regimes (reasoning).
Triplestores. Differences with graph databases. Native implementations. Implementations based on the relational data model. Impact of such decisions on the basic operations.
Distributed triplestore. Needs and difficulties. Graph Engine 1.0 as paradigm of distributed triplestore.
Main distributed algorithms.
Property and knowledge graphs comparison. Use cases
Recap about both models. Commonalities and differences. Concepts to borrow between both paradigms.
Main use cases. Metadata management: Data Lake semantification and data governance.
Main use cases. Exploitation of their topological features: recommenders on graphs and data mining.
Visualization: by means of a GUI (Gephi) or programmatically (D3.js or GraphLab).
During lectures the main concepts will be discussed. Lectures will combine master lectures and active / cooperative learning activities. The student is meant to have a pro-active attitude during active / cooperative learning activities. During master lectures, the student is meant to listen, take notes and ask questions. Objectives:32154 Contents:
Theory: These lectures comprise the teacher's explanations and constitute the main part of the course. The students will also have some contents to read and prepare outside the classroom and will be asked to participate in cooperative learning activities.
Laboratory: Mainly, the lab sessions will be dedicated to the practice (with and without computer) of the concepts introduced in the theory lectures. Specific and relevant tools will be introduced in these sessions. Small-sized projects will be conducted using these tools.
Final mark = 10% EC + 40% EX + 40% LAB + 10% C
EX = Final exam mark
LAB = Weighted mark of the labs
EC = Mark from the activities in the theoretical sessions
C = Peer evaluation
EC = In some theory sessions some activities will be proposed. The students need to solve it and hand it out to the lecturer before the end of the session.
LAB: There are three lab sessions, each one with a potential different weight. LABs will be performed in groups assigned by the lecturer.
C: During the course practice each student will interact with other students. Since the practice is meant to entail several working hours each student will peer-mark his / her teammates. The lecturer will assign a mark to each student according to the peer-marking received by his / her teammates.
The Ubiquity of Large Graphs and Surprising Challenges of Graph Processing -
Sahu, Siddhartha; Mhedhbi, Amine; Salihoglu, Semih; Lin, Jimmy; Özsu, M. Tamer,
Cornell University Library, 2017. https://arxiv.org/abs/1709.03188
The student must be familiar with basics on databases and data modeling. Programming skills are also mandatory.
Where we are
B6 Building Campus Nord
C/Jordi Girona Salgado,1-3
08034 BARCELONA Spain
Tel: (+34) 93 401 70 00