Advanced Human Languages Technologies

You are here

Home » Studies » Masters » Master in Artificial Intelligence » Curriculum » Syllabus » Advanced Human Languages Technologies
Credits
5
Types
  • MIRI: Specialization complementary (Data Science)
  • MAI: Elective
Requirements
This subject has not requirements, but it has got previous capacities
Department
CS;TSC
Web
https://www.cs.upc.edu/~padro/ahlt/ahlt.html
This course offers an in-depth coverage of main basic tasks for Natural Language Processing. We will present fundamental models and tools to approach a variety of Natural Language Processing tasks, ranging from named entity recognition to syntactic processing and document classification. The flow of the course is along two main axis: (1) computational formalisms to describe natural language processes, and (2) statistical and machine learning methods to acquire linguistic models from large data collections and solve specific linguistic tasks

Teachers

Person in charge

  • Lluis Padro Cirera ( )

Weekly hours

Theory
1.5
Problems
0.5
Laboratory
1
Guided learning
0
Autonomous learning
5.3333

Competences

Generic Technical Competences

Generic

  • CG3 - Capacity for modeling, calculation, simulation, development and implementation in technology and company engineering centers, particularly in research, development and innovation in all areas related to Artificial Intelligence.

Technical Competences of each Specialization

Academic

  • CEA3 - Capability to understand the basic operation principles of Machine Learning main techniques, and to know how to use on the environment of an intelligent system or service.
  • CEA5 - Capability to understand the basic operation principles of Natural Language Processing main techniques, and to know how to use in the environment of an intelligent system or service.

Transversal Competences

Teamwork

  • CT3 - Ability to work as a member of an interdisciplinary team, as a normal member or performing direction tasks, in order to develop projects with pragmatism and sense of responsibility, making commitments taking into account the available resources.

Reasoning

  • CT6 - Capability to evaluate and analyze on a reasoned and critical way about situations, projects, proposals, reports and scientific-technical surveys. Capability to argue the reasons that explain or justify such situations, proposals, etc..

Analisis y sintesis

  • CT7 - Capability to analyze and solve complex technical problems.

Basic

  • CB6 - Ability to apply the acquired knowledge and capacity for solving problems in new or unknown environments within broader (or multidisciplinary) contexts related to their area of study.
  • CB8 - Capability to communicate their conclusions, and the knowledge and rationale underpinning these, to both skilled and unskilled public in a clear and unambiguous way.
  • CB9 - Possession of the learning skills that enable the students to continue studying in a way that will be mainly self-directed or autonomous.

Objectives

  1. Learn to apply statistical methods for NLP in a practical application
    Related competences: CEA3, CEA5, CEA7, CEA10, CT3, CB6, CB8,
  2. Understand statistical and machine learning techniques applied to NLP
    Related competences: CEA3, CG3, CEP3, CT6, CT7, CB6,
  3. Develop the ability to solve technical problems related to statistical and algorithmic problems in NLP
    Related competences: CEA3, CEA5, CEA7, CEA10, CG3, CT7, CB6, CB8, CB9,
  4. Understand fundamental methods of Natural Language Processing from a computational perspective
    Related competences: CEA5, CEP4, CT7, CB6,

Contents

  1. Statistical Models for NLP
    Introduction to statistical modelling for language. Maximum Likelhood models and smooting. Maximum entropy estimation. Log-Linear models
  2. Distances and Similarities
    Distances (and similarities) between linguistic units. Textual, Semantic, and Distributional distances. Semantic spaces (WN, Wikipedia, Freebase, Dbpedia).
  3. Sequence Predicion
    Prediction in word sequences: PoS tagging, NERC. Local classifiers, HMM, global predictors, Log-linear models.
  4. Syntactic Parsing
    Parsing constituent trees: PCFG, CKY vs Inside/outside
    Parsing dependency trees: CRFs for parsing. Earley algorithm
  5. Word Embeddings
    Static word embeddings. Word2Vec, Glove
    Limitations - need of contextual embeddings
  6. Recurrent Neural Networks
    RNNs for Language Modeling and sequence labeling
    Bottleneck problem. LSTMs
    Vanishing gradient problem
    LSTM-based word embeddings: ELMO
  7. Convolutional Neural Networks
    CNNs for NLP. 1D kernels vs 2D kernels.
    stride, padding
    Pooling
    NLP tasks suitable for CNNs vs RNNs
  8. Transformers
    Vanishing gradient problem in RNN/LSTM
    Attention
    Tranformer architecture
  9. Large Language Models
    Large Language Models: origin and evolution
    Reinforcement Learning from Human Feedback
    Fundational vs Instructed LLMs
    Use of LLMs in NLP applications: Zero shot, few-shot, fine-tuning
    Optimization and efficiency issues
  10. Ethics - Limitations and Risks of LLMs
    Biases
    Hallucinations
    Security
    Environmental costs
    Social costs

Activities

Activity Evaluation act


Statistical Language Models

Introduction to statistical modelling for language. Maximum Likelhood models and smooting. Maximum entropy estimation. Log-Linear models
Objectives: 4 2
Contents:
Theory
2.5h
Problems
0.5h
Laboratory
0h
Guided learning
0h
Autonomous learning
0h

Distances and Similarities

Distances (and similarities) between linguistic units. Textual, Semantic, and Distributional distances. Semantic spaces (WN, Wikipedia, Freebase, Dbpedia). Latent Semantic Analysis. Word Embeddings
Objectives: 4 2
Contents:
Theory
2h
Problems
0.5h
Laboratory
0h
Guided learning
0h
Autonomous learning
0h

Sequence Prediction

These lectures will present sequence labeling models, an important set of tools that is used for sequential tasks. We will present this in the framework of structured prediction (later in the course we will see that the same framework is used for parsing and translation). We will focus on machine learning aspects, as well as algorithmic aspects. We will give special emphasis to Conditional Random Fields.
Objectives: 4 2
Contents:
Theory
2h
Problems
1h
Laboratory
4h
Guided learning
0h
Autonomous learning
8h

Syntax and Parsing

We will present statistical models for syntactic structure, and in general tree structures. The focus will be on probabilistic context-free grammars and dependency grammars, two standard formalisms. We will see relevant algorithms, as well as methods to learn grammars from data based on the structured prediction framework.
Objectives: 4 2
Contents:
Theory
3h
Problems
1h
Laboratory
4h
Guided learning
0h
Autonomous learning
8h

Word Embeedings

Static Word embeddings. Word2Vec, Glove, FastText
Objectives: 4 2
Contents:
Theory
2h
Problems
0.5h
Laboratory
0h
Guided learning
0h
Autonomous learning
0h

Recurrent Neural Networks

Recurrent Neural Networks. Bottleneck problem. LSTMs. Vanishing Gradient problem
Objectives: 4 2
Contents:
Theory
2h
Problems
1h
Laboratory
4h
Guided learning
0h
Autonomous learning
8h

Convolutional Neural Networks

CNNs for NLP. 1D kernels vs 2D kernels stride, padding, pooling
Objectives: 2
Contents:
Theory
2h
Problems
1h
Laboratory
3h
Guided learning
0h
Autonomous learning
6h

Transformers

Attention. Transformers
Objectives: 3
Contents:
Theory
2h
Problems
1h
Laboratory
0h
Guided learning
0h
Autonomous learning
3h

Large Language Models

Large Language Models. Origin & Evolution. Usage: zero-shot, few-shot, fine tuning optimization
Objectives: 3 1
Contents:
Theory
3h
Problems
1h
Laboratory
0h
Guided learning
0h
Autonomous learning
2h

Lab project


Objectives: 3 1
Week: 15 (Outside class hours)
Theory
0h
Problems
0h
Laboratory
0h
Guided learning
0h
Autonomous learning
30h

Final exam



Week: 15 (Outside class hours)
Theory
2h
Problems
0h
Laboratory
0h
Guided learning
0h
Autonomous learning
15h

Teaching methodology

The course will be structured around four different linguistic analysis levels: word level, phrase level, sentence level, and document level. Typical NLP tasks and solutions corresponding to each level will be presented.
The first half of the course is devoted to "classical" statistical and ML approaches. The second half of the course revisits the same levels under a deep learning perspective

Theoretical background and practical exercises will be developed in class.

Finally, students will develop a practical project in teams of two students. The goal of the project is to put into practice the methods learned in class, and learn how the experimental methodology that is used in the NLP field. Students have to identify existing components (i.e. data and tools) that can be used to build a system, and perform experiments in order to perform empirical analysis of some statistical NLP method.

Evaluation methodology

Final grade = 0.5*FE + 0.5*LP

where

FE is the grade of the final exam

LP is the grade of the lab project

Bibliography

Basic:

Web links

Previous capacities

- Although not mandatory, familiarity with basic concepts and methods of Natural Language Processing is strongly recommended

- Good understanding of basic concepts and methods of Machine Learning.

- Advanced programming skills.