• Course code:63835E
  • Contents

Tensor Networks for Machine Learning

Tensor networks are decompositions of multi-dimensional tensors with exponential reduction of parameters. They have been introduced to quantum mechanics approximately 20 ago. Since then, they have become one of the most important technical tools to understand quantum states’ structure, especially in one dimension, and a vital ingredient of the state-of-the-art numerical techniques of many-body quantum mechanics. In the field of many-body quantum mechanics and quantum information, tensor networks are now a well established and understood tool with well known geometric properties and robust optimization algorithms. In the last seven years, they also appeared in mathematical literature (in particular matrix product states or tensor trains) in linear algebra with large matrices. Over the previous four years, they increasingly appear in machine learning literature, where they have been applied to a variety of practical problems from parameter compression, classification to anomaly detection. Theoretically, tensor networks have been related to Born machines, hidden Markov models, and probabilistic automata and quadratic automata from the formal language’s literature. The course will focus on recently developed tensor networks applications to machine learning (mainly from an experimental/numerical perspective).

The general idea is to guide students in reproducing recent results involving tensor network decompositions in machine learning and then trying to go beyond by improving the techniques or applying the learned techniques to a slightly different problem. The proposed projects can be adapted to fit student interests, time, and expertise.

Possible topics:

  1. Tensor networks for sequence (language) modeling
    Tensor networks have been successfully applied to model regular expressions. The student should revisit this problem, reproduce the main results and potentially apply the techniques to context-free languages or other sequence modelling tasks.

    Reference: Miller, Jacob, Guillaume Rabusseau, and John Terilla. "Tensor networks for probabilistic sequence modeling." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.
  2. Neural network compression
    Tensor networks can exponentially reduce the number of parameters in neural network layers, especially fully connected layers. Several efficient compressions have already been demonstrated. The student should reproduce the compression results and contrast them with different compression techniques. A motivated student can apply the methods to big language models (e.g. BERT or GPT2).

    Reference: Novikov, Alexander, et al. "Tensorizing neural networks." arXiv preprint arXiv:1509.06569 (2015)
  3. Anomaly detection
    Tensor networks seem to be an effective tool for anomaly detection. State-of-the-art results have recently been obtained on several tabular datasets. The student should reproduce those results and go beyond the current state-of-the-art on a chosen dataset by applying a more sophisticated augmentation, parameter search, different architectures.

    Reference: Wang, Jinhui, et al. "Anomaly Detection with Tensor Networks." arXiv preprint arXiv:2006.02516 (2020)
  4. Positive unlabeled learning
    Reproduce the tensor network approach to anomaly detection and adapt it for the positive unlabeled learning problem.

    Reference: Wang, Jinhui, et al. "Anomaly Detection with Tensor Networks." arXiv preprint arXiv:2006.02516 (2020)
  5. Generative modelling
    Tensor networks have been successfully applied to generative modelling of MNIST images without considering the geometry of the images. The student should first reproduce the results on generative modelling with matrix product states. If successful, two additional tasks can be tackled:
    • Generative modelling of images with 2D tensor networks
    • Generative modelling of tabular data, in particular medical data

      Reference: Han, Zhao-Yu, et al. "Unsupervised generative modeling using matrix product states." Physical Review X 8.3 (2018): 031012
  6. Advanced topics
    Several more advanced topics are also available, e.g., adversarial examples (theory and experiment) for tensor network models, non-linear tensor networks, scale-invariant layers with uniform tensor networks, entanglement and dataset symmetries, and tensor network algorithms for quantum computers.
  • Study programmes