TryAlgo

LLMs and RAG in education

2025-10-27T00:00:00+00:00

This summer, I was in Palermo where the Educational Data Mining, Learning at Scale & AI in Education conferences were held.

I presented Anav Agrawal’s RAG tool, which we use in our course.

Anav Agrawal, Jill-Jênn Vie. AlgoAce: Retrieval-Augmented Generation for Assistance in Competitive Programming. CSEDM 2025 - 9th Educational Data Mining in Computer Science Education Workshop, Jul 2025, Palermo, Italy. [paper] [code]

I’ll tell you about what some colleagues from Cornell University and Berkeley University presented because it’s truly impressive.

Cornell

The professor (Rene Kizilcec) has 270 students.
He uploads his PDFs to some website hita.ai developed by former students.
Students can ask questions anonymously (it sources the answers to the pages of the course slides or to specific points in a course video) and the professor sees the anonymous conversations (this part is the most valuable).
He has automatic analytics on the most frequently asked questions.
The (6k?) students have asked 64k questions on the platform (local LLM) in a few years.
The professor uses it to verify that students are actually reading the articles he assigns (students have to debate a question with an AI that has read the article).

He told the anecdote that his colleague noticed that 5 students were cheating and told them: “Wow, I’m really impressed by the richness of your reasoning, how about you come and discuss it in my office?” and then the students said “Nooo, sorry, we cheated.”

Berkeley

Narges Norouzi has 900-1700 students per cohort in their first CS1 course, required for all students in the computer science and engineering departments.
Students are allowed to use an autograder for their homework but not for graded labs or projects https://sp25.datastructur.es/policies/ see also, from a different u, https://eecs-autograder.github.io/autograder.io
They reuse the (1 million) code submissions from students over the previous (7) years to help students debug, see what types of hints are useful, which directly feeds into their (the professors’) research. They have 105,000 queries from 2,000 students from the 2023-2024 academic year to their local LLM.
They are not allowed to conduct randomized controlled trials; their ethics committee (IRB) prohibits it, due to the impactness of this course and its scale.
They show that students complete homework faster with AI (unsurprisingly) but not practical assignments faster (they even have some negative results on practical assignments).
They had two papers nominated for best paper at AIED 2025 (below), one of which uses the curriculum to automatically determine students’ progress in the course (knowledge tracing) and make appropriate recommendations.

Modeling Student Knowledge Progression in Intelligent Tutoring Interactions Abigail O’Neill, Kanav Mittal, Hanna Schlegel, Gireeja Ranade and Narges Norouzi https://drive.google.com/file/d/1As0EAEXOeyTnDqaMCOQqYY__BaN_d9gq/view?usp=sharing

Askademia: A Real-Time AI System for Automatic Responses to Student Questions Gaurav Tyagi, Meenakshi Mittal, Azalea Bailey, Gireeja Ranade and Narges Norouzi https://drive.google.com/file/d/1TOAuYZutWz8IWmWj_NLneL8atbUEcoVK/view?usp=drive_link

Différence entre mémoïsation et programmation dynamique

2025-09-03T00:00:00+00:00

Update : Merci à Marc de Falco qui a trouvé une erreur dans le code initial. J’élabore là-dessus ci-dessous.

L’autre jour, en jury de l’agrégation d’informatique où j’ai eu la chance de participer de 2022 à 2025 en compagnie de collègues et candidats aussi brillants que passionnés, le candidat choisit la leçon sur la programmation dynamique et fait son développement sur le célèbre algorithme de sac à dos. Une collègue pose la question : “Qu’est-ce qui se passe si les capacités des poids sont négatives ?”

Cette question permet d’illustrer une différence entre mémoïsation et programmation dynamique.

On commence par la façon classique de résoudre le problème dans le cas des capacités positives $c_i \geqslant 0$. La relation de récurrence (équation de Bellman) est la suivante.

Si $maxval[i][c]$ est la capacité maximale que l’on peut atteindre avec les $i \geqslant 0$ premiers objets et une capacité limite de $c$, alors $maxval[0][c] = 0$ pour toute capacité $c$ (pas d’objet) et quand $i \geqslant 1$,

\[maxval[i][c] = \max \begin{cases}maxval[i - 1][c] \textrm{ (si on ne prend pas le $i$-ème objet)}\\maxval[i - 1][c - c_i] + v_i \textrm{ si $c \geqslant c_i$ (si on prend le $i$-ème objet)}\end{cases}\]

On peut faire un balayage avec une double boucle pour aller en nombre d’objets croissant et capacité croissante (bottom-up). Complexité temps et mémoire : $O(nC)$ où $n$ est le nombre d’objets et $C$ est la capacité du sac à dos.

capacities = [2, 3, 5]
values = [6, 4, 2]
cmax = 9

def knapsack():
    n = len(capacities)
    max_value = [[0] * (1 + cmax) for _ in range(n + 1)]
    candidates = []
    for i in range(1, n + 1):
        for c in range(0, cmax + 1):
            candidates = [max_value[i - 1][c]]
            if c >= capacities[i - 1]:
                candidates.append(values[i - 1] + max_value[i - 1][c - capacities[i - 1]])
            max_value[i][c] = max(candidates)
    return max_value[n][cmax]

knapsack()

En effet, on peut obtenir une valeur de 10 avec les 2 premiers objets (capacité totale 2 + 3 = 5 en dessous de la limite 9).

Cette méthode nous permet de répondre à toutes les questions possibles : que vaut $maxval$ pour les $i$ premiers objets et une capacité limite $c$ ? On visite tous les états possibles $(i, c)$.

Regardons à présent à quoi ressemblerait une solution qui mémoïserait (i.e. stockerait les appels récursifs pour ne pas recalculer la même quantité plusieurs fois). On peut s’appuyer sur l’astuce Python du décorateur @cache. (On est plusieurs profs à penser que la mémoïsation devrait être enseignée avant la programmation dynamique.) Notez au passage à quel point ce code est similaire au précédent. En particulier, si on faisait du Haskell, on n’aurait pas besoin du décorateur.

from functools import cache

capacities = [2, 3, 5]
values = [6, 4, 2]
cmax = 9

@cache
def max_value(i, c):
    candidates = []
    if i == 0:
        return 0
    candidates.append(max_value(i - 1, c))
    if c >= capacities[i - 1]:
        candidates.append(values[i - 1] + max_value(i - 1, c - capacities[i - 1]))
    return max(candidates)

n = len(capacities)
max_value(n, cmax)

Complexité : autant d’appels que strictement nécessaire, $O(nC)$ au pire. Mais soyons plus précis.

Soit $C_M$ le nombre d’états uniques (cases mémoire) parcourus par l’algorithme. On a d’une part $C_M \leq (n + 1)(C + 1)$ car c’est le nombre total d’états possibles, et d’autre part $C_M \leq 2^n$ car c’est le nombre total de combinaisons possibles (et d’appels récursifs si on ne faisait pas de mémoïsation).

On voit bien que selon le régime, par exemple si $n$ est petit et $C$ grand, il vaut mieux tout tester, même avec l’algorithme naïf, plutôt que de remplir tous les états possibles. Si $C$ est petit on a plutôt envie d’exploiter la redondance. Dans tous les cas, l’algorithme de mémoïsation est optimal en temps (mais pas en mémoire). Le gain par rapport au naïf est $C_M / 2^n$ et par rapport à l’algo de programmation dynamique est $C_M / ((n + 1)(C + 1))$. Faudrait que je fasse un dessin sur une instance avec les cases visitées pour chaque algorithme.

À présent, regardons ce qui se passe dans le cas de capacités négatives. (Veillez à bien exécuter les cellules précédentes avant celle-ci.)

~~La mémoïsation fonctionne et reste optimale.~~ Update : en fait non, le code ci-dessus est faux. Cherchez un contre-exemple, en voici un.

capacities = [2, -1]
values = [1, 1]
cmax = 1

n = len(capacities)
max_value.cache_clear()  # Sinon les appels sont encore dans la mémoire
max_value(n, cmax)

Ici, la réponse est bonne : 2, on prend tous les objets. Mais si on met les objets dans l’autre ordre :

capacities = [-1, 2]
values = [1, 1]
cmax = 1

n = len(capacities)
max_value.cache_clear()
max_value(n, cmax)

On rate l’objet de poids 2 car on ne s’autorise pas à passer par des états de capacités négatives qu’on pourrait rattraper plus tard. Autorisons donc les capacités négatives tant que l’état final convient. La relation de récurrence est donc :

\[maxval[0][c] = \begin{cases}0 \textrm{ si $c \geqslant 0$ (il reste de la capacité)}\\ -\infty \textrm{ sinon}\end{cases}\\ maxval[i][c] = \max \begin{cases}maxval[i - 1][c] \textrm{ si on ne prend pas le $i$-ème objet}\\maxval[i - 1][c - c_i] + v_i \textrm{ si on prend le $i$-ème objet}\end{cases}\]

Code modifié :

from functools import cache

capacities = [-1, 2]
values = [1, 1]
cmax = 1

@cache
def max_value_neg(i, c):
    candidates = []
    if i == 0:
        return 0 if c >= 0 else float('-inf')  # Valeur admissible s'il reste de la capacité
    return max(
        max_value_neg(i - 1, c),
        values[i - 1] + max_value_neg(i - 1, c - capacities[i - 1])
    )

n = len(capacities)
max_value_neg(n, cmax)

Quelle est sa complexité ? On pourrait l’améliorer en éliminant les branches non prometteuses (s’il ne reste plus d’objets pour aboutir à un état positif). Qu’en est-il de l’algorithme itératif de programmation dynamique au début de ce billet ?

capacities = [-1, 2]
values = [1, 1]
cmax = 1

knapsack()

Exercice au lecteur : comment modifier le code précédent de knapsack pour qu’il fonctionne dans le cas de capacités négatives ? Quel est le nombre d’états maximaux à considérer ? Quelle est la structure de données adaptée ? Faites un dessin sur papier pour voir ce qui se passe.

Une autre solution d’un étudiant est de partir d’un état où on a pris toutes les capacités négatives et considérer des actions “enlever l’objet” afin de se ramener au sac à dos à capacités positives. C’est également une solution correcte.

Méta : si ça vous intéresse de voir comment j’ai chargé pyodide sur ce blog post Jekyll, vous pouvez regarder en bas du source de cette page. Vous pouvez faire une PR pour remplacer le premier bloc par un textarea (avec coloration syntaxique svp) pour modifier le code.

Gamme pythagoricienne

2024-11-29T00:00:00+00:00

En musique, on change d’octave en doublant la fréquence. On passe à la quinte en multipliant par 3/2.

Donc pour passer au demi-ton suivant il suffit de multiplier la fréquence par $2^{1/12}$. En Python :

for i in range(12):
    print(i, 220 * (2 ** (i / 12)))

Ah oui mais c’est embêtant ça car on sait que la quinte parfaite est à 3/2 ce qui donnerait un mi à 330 Hz et non à 329.63 Hz (cf. ligne 7).

Du coup, soit tous les pianos sont faux, soit les demi-tons ne sont pas égaux. La réponse est la première.

To know more check this amazing presentation from Kenya Otsuka (Kyoto University):

Longest increasing subsequence

2024-09-28T00:00:00+00:00

If we want to find the longest (strictly) increasing subsequence of an array $a$ of size $n$, of course we can assume that $dp[i]$ is the answer for the first $i$ elements and then, as a LIS of size $n$ contains a LIS of size $n - 1$:

\[dp[i] = \max_{\substack{j < i\\ a_j < a_i}} dp[j] + 1\]

This gives a first algorithm in $O(n^2)$. Can we do better?

We do not need to remember all LIS of size $\ell$. We just need to remember the smallest end for a LIS of size $\ell$, called $t[\ell]$.
The list of smallest ends happens to be increasing (but not necessarily a subsequence). Each $t[\ell]$ forces $t[\ell - 1]$ to be lower.

In this case, we can binary search for the opt LIS to which we can add one element. This gives $O(n \log n)$. I think this is an example of Divide and Conquer DP.

I was wondering what was the $O(n \log n)$ that relies on a segment tree. A blog post on Codeforces had the answer (of course!). We need to define a new dp:
$dp[i, v]$ is the LIS using first $i$ elements finishing in $v$. Then we just need to do a min query over $dp[i, 1:v - 1]$.

(A notebook is on the way.)

Aho-Corasick

2024-09-11T00:00:00+00:00

Étant donnée une liste de chaînes de caractères L, construire une structure de données qui permet en temps linéaire d’identifier toutes les occurrences des mots de L à l’intérieur d’un mot S donné.

Note

(dernière mise à jour de ce billet: 5 novembre 2024)

Ce billet est inspiré d’une page similaire du site web excellent CP-Algorithms, qui a son tour est une traduction d’un site web russe très complet.

Dans ce billet nous allons utiliser de manière interchangeable les termes chaîne, chaîne de caractères et mot.

Idée clé

L’algorithme de Knuth-Morris-Pratt construit en quelque sorte un automate à partir d’un mot W donné, et permet en temps linéaire d’identifier toutes les occurrences de W comme sous-chaîne d’une chaîne S donnée. Plutôt que de construire un automate indépendant pour chaque mot dans L, on va construire un seul automate pour tout l’ensemble des mots dans L.

Arbre préfixe

Le premier ingrédient est de construire un arbre préfixe (aussi appelé une trie), pour tous les mots dans L. L’arbre est enraciné avec des arcs qui pointent partant de la racine. Les arcs sortant d’un nœud sont étiquetés par des lettres distinctes. Ainsi on associe à un nœud un mot qui est la concaténation des lettres le long du chemin de la racine vers ce nœud. Si le mot est dans L, alors cette propriété est stockée dans un attribut pattern du nœud. Dans notre implémentation, pattern sera l’indice du mot dans L, ou la constante -1 le cas échéant.

Dans l’illustration ci-dessous, les nœuds correspondants à des mots dans L={i,in,tin,sting} sont montrés avec un contour double. Les arcs rouges et bleus seront expliqués par la suite.

Liens suffix

Un tel arbre va être utilisé comme un automate. En partant de la racine, on va suivre les liens sortants tel qu’indiqués par les lettres successives dans S.

Supposons que l’exécution de l’automate a atteint un sommet v, après avoir traité un préfixe T de S, et que la prochaine lettre est c. Malheur: aucun arc sortant n’est étiqueté par c. Que faire ? Il nous faut bien aller vers un sommet de l’arbre, mais lequel ?

L’idée est que nous cherchons à maintenir l’invariant suivant. À chaque sommet u de l’arbre correspond un mot, tel que défini par le chemin de la racine vers u. Certains de ces mots pourraient être des suffixes de T. Et le mot associé à v est le plus long de ces suffixes.

Alors comme il n’y pas d’arc avec c qui sort de v, on doit se rabattre sur un suffixe plus court. Soit w le mot associé au nœud v. On cherche alors le sommet u qui correspond au plus long suffixe strict de w. Strict veut dire qu’il n’est pas w lui-même, mais plus court.

L’arbre est augmenté avec un lien suffixe partant de tout sommet, excepté la racine, qui dans cet exemple va de v à u. Et on va suivre ces liens, jusqu’à ce qu’on tombe sur un sommet qui aurait un arc sortant étiqueté par la lettre c, au pire de cas on remonte vers la racine. Puis on fait la transition habituelle par la lettre c à partir de ce sommet.

On pourrait penser alors que le traitement d’une chaîne $s$ ne se fait pas en temps linéaire en la longueur de $s$, car pour une lettre donnée on doit suivre peut-être plusieurs liens suffixe. Imaginons que nous observons dans $s$ une fenêtre. La sous-chaine de $s$ dans la fenêtre est le mot qui correspond au nœud courant dans la trie. Quand nous traitons la lettre suivante dans $s$, nous agrandissons la fenêtre vers la droite d’une position, et quand nous remontons avec un lien suffix dans la trie, alors nous diminuons la fenêtre de la gauche par au moins une position. Les deux bords de la fenêtre ne vont que vers la droite et leur nombre total de déplacements est majoré par deux fois la longueur de $s$.

Mais comment calculer ces liens suffixes? Considérons un arc $u\rightarrow v$ dans la trie, étiqueté par la lettre $c$. En remontant les liens suffixes de $u$, on trouve le premier sommet $u’$, avec un arc sortant $u’\rightarrow v’$ qui est également étiqueté par la lettre $c$. Confondons pour cette explication les nœuds de la trie avec le mot auxquels ils correspondent. Comme $u’$ est un suffixe strict de $u$, on a également que $u’c$ est un suffixe strict de $uc$. Et comme $u’$ est le premier nœud rencontré avec un arc sortant étiqueté par $c$, la chaîne $u’c$ aussi le plus grand des suffixes de $uc$ qui correspond à un nœud dans la trie.

Dans l’illustration ci-haut, les arcs suffixes sont montrés en rouge.

Liens output

Voici une première procédure de recherche.

Construire la trie et les liens suffixes
v = racine
pour toute lettre c dans la chaîne s:
    # avancer dans la trie
    tant qu'il n'existe pas d'arc sortant v -> v' étiqueté par c:
        v = v.suffix
    v = v'
    si v est marqué comme correspondant à un des motifs à trouver:
        annoncer une occurrence de ce motif dans s

Si on applique cette procédure avec l’exemple donnée sur la chaîne stingin, alors on trouve bien les occurrences

stingtin
sting
     tin

mais on a raté d’autres occurrences de motifs, comme par exemple i et in. Raison est que certains motifs apparaissent dans d’autres. Alors il faut que pendant le parcours dans la trie, à chaque fois qu’on est dans un nœud v, qu’on trouve tous les suffixes de v (incluant v lui-même), qui soit un des motifs à détecter. Pour cela il suffit de remonter les liens suffixes à partir de v et annoncer une occurrences pour chacun les nœuds rencontrés marqués comme étant des motifs à détecter. Pour accélérer cette recherche, on crée des raccourcis vers le prochain suffix marqué. Nous illustrons ces raccourcis par des arcs bleus qu’on appelle liens output.

Par exemple dans la trie ci-haut, il y a un lien output de sti directement vers i.

Complexité

La construction de la trie se fait en temps O(mk) où m est la longueur totale des mots dans L, et k est la taille de l’alphabet.

La recherche des occurrences des mots de L dans un mot S se fait en temps linéaire en la longueur de S et dans le nombre d’occurrences détectées.

Détails d’implémentation

Les arcs sortant sont implémentés par un tableau next. À la place des lettres on travaille en interne avec leur rang, avec un décalage tel que la plus petite lettre a le rang 0. La constante LOW donne le code Ascii de la plus petite lettre et LEN donne la taille de l’alphabet. Les codes Ascii des lettres dans l’alphabet doivent se suivre sans interruption.

Pour construire les liens suffixe on fait un parcours en largeur de la trie. Ainsi on aura traité les niveaux précédant un nœud au moment de le traiter. Ceci se fait avec une file Q, contenant des sommets à traiter. Détail important: en même temps que de construire les liens suffix et output, nous complétons également les liens next[i] qui seraient encore None. Ainsi nous construisons un automate complet. Autrement au moment de chercher les motifs dans une chaîne donnée on peut juste suivre les liens données par next pour chaque lettre de la chaîne.

from collections import deque 

class Vertex:
    """Vertex of the Aho-Corasick trie
    """

    LOW = ord('a')  # Ascii code of smallest letter in the alphabet
    LEN = 26        # Alphabet size

    def index(ch):  # transforms the character into internal index
        i = ord(ch) - Vertex.LOW
        assert 0 <= i < Vertex.LEN
        return i

    def __init__(self):
        self.pattern = -1 # -1 means no pattern, otherwise it is the pattern index
        self.next   = [None] * Vertex.LEN
        self.suffix = None
        self.output = None    

class Aho_Corasick:

    def __init__(self, patterns):
        """patterns is a list of strings. 
        Later when we search for them, we return only their index in that list 
        and the matching position.
        
        :complexity: O(sum(map(len, patterns)))
        """
        # 1. build the trie
        self.root = Vertex()
        for pattern_index, p in enumerate(patterns):        
            # add pattern p to the trie
            v = self.root   # current vertex in the tree ...
            for ch in p:    # ... corresponding to a prefix of p
                i = Vertex.index(ch)
                if v.next[i] is None:   # create vertices on the fly
                    v.next[i] = Vertex()
                v = v.next[i]           # descend in the trie
            v.pattern = pattern_index   # mark that this is a pattern
        # 2. augment with suffix and output links
        # and turn it into a full automaton
        Q = deque()               # queue of vertices to be processed
        # start with the root
        for i, v in enumerate(self.root.next):     
            if v is None:
                self.root.next[i] = self.root
            else:
                v.suffix = self.root 
                Q.appendleft(v)             # initially Q contains the first level vertices
        while Q:
            u = Q.pop()                     # process u
            for i, v in enumerate(u.next):  # all arcs u -> v labeled i
                if v is None:
                    u.next[i] = u.suffix.next[i]
                else:
                    Q.appendleft(v)
                    v.suffix = u.suffix.next[i]
            if u.suffix.pattern != -1:
                u.output = u.suffix
            else:
                u.output = u.suffix.output 
                
    def match(self, s):
        """ find all substrings of s which are among the stored strings.
            Iterates over positions in s and the matched string ending at this position.
        """
        v = self.root               # current vertex
        for j, ch in enumerate(s):
            i = Vertex.index(ch)    # descend one step
            v = v.next[i] 
            a = v                   # now output all found patterns
            while a is not None:
                if a.pattern != -1: # this test can fail only for v itself
                    yield(j, a.pattern)
                a = a.output        # follow the output links

if __name__ == "__main__":
    patterns = ["i", "in", "tin", "sting"]
    AC = Aho_Corasick(patterns)
    s = "istingin"
    print(s)
    for e, p in AC.match("istingin"):
        k = len(patterns[p])
        print(" "*(e-k+1) + patterns[p])

Problèmes

Attention, le code Python est trop lent pour ces problèmes.

Approximations of the Euclidean metric traveling salesman problem

2024-05-12T00:00:00+00:00

Back in the good ol’ agrégation days, I remember I used as développement¹ a nice 2-approx algorithm for the traveling salesman problem where the weights on the edges are given by the Euclidean distance between nodes.

I was told it was Christofides algorithm but actually it was not. It is the “double-tree algorithm”.

Find a minimum spanning tree $T$ using e.g. Kruskal’s algorithm.
Duplicate the edges of $T$. Find an Eulerian tour (that exists) using e.g. Hierholzer’s algorithm.
Shortcut the Eulerian tour. This is a 2-approx of the (Euclidean) metric TSP.

It was hard to find who discovered it but Rozenkrantz et al. say that it is a “widely known but unpublished method” (1977). Christofides and Serdyukov found in 1976 (published in 1978) that by solving a matching problem between nodes of odd order (at the cost of $O(n^3)$), they could improve the approximation ratio to 3/2. This reminds me of the trick used by ENS Ulm team in Google Hash Code 2014.

To know more, you can check this other post.

Update. Karlin, Klein and Gharan found a new algorithm with approximation ratio $3/2 - 10^{-36}$, and got the best paper at STOC 2021.

References

Rosenkrantz, Daniel J., Richard E. Stearns, and Philip M. Lewis, II. “An analysis of several heuristics for the traveling salesman problem.” SIAM journal on computing 6.3 (1977): 563-581.

Christofides, N. “Worst-case analysis of a new heuristic for the traveling salesman problem.” Symp. on New Directions and Recent Results in Algorithms and Complexity (April 1976), Carnegie-Mellon University, Pittsburgh

Serdyukov, Anatoliy (1978), “О некоторых экстремальных обходах в графах” [On some extremal walks in graphs], Upravlyaemye Sistemy (Управляемые системы) (in Russian), 17: 76–79

Karlin, Anna R., Nathan Klein, and Shayan Oveis Gharan. “A (slightly) improved approximation algorithm for metric TSP.” Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing. 2021. https://arxiv.org/abs/2007.01409

This must mean nothing to non-French people but anyway. ↩

Count particular rectangles in a matrix

2024-03-27T00:00:00+00:00

Given a matrix with distinct values, a rectangle consists of 4 cells at the intersection of two distinct rows and two distinct columns. It is good if the largest 2 values of the 4, are on the same row or the same column. Count the number of good rectangles in linear time, in the size of the matrix.

Discussion

This is a problem from the warmup competition of ICPC/European Championship 2024. During a discussion between Christoph, Pavel and Aris, the following elegant solution was found by Pavel. Once you have the key idea, the solution is quite simple to implement. The trick is that instead of counting good rectangles we count good triangles. There is a linear equation relating these two quantities.

Key idea

Consider the following illustration of the problem, depicting 3 particular rectangles as an example. The values of the 4 cells are replaced by their rank. The first two rectangles are good, because the two largest cells, of rank 3 and 4, are in the same row or the same column. The third rectangle is bad, because the two largest cells are diagonally opposite.

Now we introduce a different object, namely a triangle. A triangle with corner C, consists of two other cells : a cell A, in the same column as C and a cell B in the same row as C. If the value of C is larger than the values of A and of B, then the triangle is called good.

A rectangle contains 4 triangles, one for each corner cell. If the corner cell has rank 1 or 2, then the triangle is bad. If the corner cell has rank 4, then the triangle is good. And the key observation is:

The triangle with corner cell of rank 3 in the rectangle, is good if and only if the rectangle is bad.

From triangles to rectangles

The matrix has dimensions $n\times n$. Let $N=n(n-1)$. Then we have the following numbers.

The number of good rectangles is denoted $R$.
The number of bad rectangles is denoted $B$.
The number of pairs of rows is $N/2$. And so is the number of pairs of columns.
The number of rectangles is $N^2 / 4$.
We have $R+B = N^2 / 4$.
The number of good triangles is denoted $T$.
We have $T=R+2B$.
Hence $R = N^2/2-T$.

Count the number of good triangles

Process the cells of the given matrix $M$ in order of increasing rank. Maintain a counter for every row and every column of the number of its already processed cells. When processing a cell, the number of triangles with this corner cell is simply the product of the counters for the corresponding row and column.

Complexity

Using bucket sort, we can process the cells of $M$ in order, in linear time. The overall running time is linear in the size of $M$.

Suffix Array

2024-03-01T00:00:00+00:00

Given a string s, sort all cyclic shifts of s. Formally produce a table p such that p[j]=i if s[i:]+s[:i] has rank j among all cyclic shifts.

Example

On input “bobocel”, here are all cyclic shifts, together with by how much they are shifted.

bobocel 0
obocelb 1
bocelbo 2
ocelbob 3
celbobo 4
elboboc 5
lboboce 6

And the same list, but lexicographically sorted. Together with the rank in the new order.

bobocel 0
bocelbo 2
celbobo 4
elboboc 5
lboboce 6
obocelb 1
ocelbob 3

Hence on this example we should output p=[0, 2, 4, 5, 6, 1, 3].

Complexity

The naïve algorithm for this problem stores all cyclic shifts in a list and sorts it. This takes time $O(n^2 \log n)$, because the list has size $n$, and comparing two strings of length $n$ takes time $O(n)$.

The problem can be solved in time $O(n)$, under some conditions on the alphabet. But we present an $O(n \log^2 n)$ implementation, which is good enough for most programming contests.

The key operation

The algorithm relies on a simple sorting function sort_class, which not only sorts a given string or list s, but also returns additional informations. It returns two tables p and c such that

p[j]=i if s[i] has rank j in sorted(s).
c[i]=j if s[i] has rank j in sorted(set(s)).

Note that the second table groups identical elements in s, and gives a rank only to the equivalence classes. For example

index   =  0123456
input s = "bobocel"

after sorting s we have for example (because identical letters can be ordered arbitrarily within each other)

original index =  0245613
sorted s       = "bbceloo"

If we rank all distinct letters of the input string we have

rank           = 0 1 2 3 4
sorted(set(s)) = b c e l o

Hence our function returns

p = [0, 2, 4, 5, 6, 1, 3]
c = [0, 4, 0, 4, 1, 2, 3]
s =  b  o  b  o  c  e  l  # for comparison

The cyclic version

We present here the cyclic version of the problem. If we want to sort the suffixes of a given string s, then we can just sort the cyclic shifts of s + special, where special is a dummy character, smaller than all characters in s.

Iteration

The idea is that for K being every integer power of 2, we want to sort the K-lengths prefixes of all cyclic shifts. For example for K=2 and s=”bobocel” we want to sort the following strings.

bo
 ob
  bo
   oc
    ce
     el
      lb

But we already have the order and equivalence classes of all (K/2)-length prefixes of all cyclic shifts. Now every K-lengths prefix is the concatenation of two (K/2)-length prefixes, say xy. Let i be the equivalence class of x and j be the equivalence class of y. Then the pair (i,j) represents the equivalence class of xy. We can use sort_class to translate the pairs into rank integers.

The strings of the example above correspond to the following pairs, where the table c from the previous iteration makes the correspondance.

(0, 4)
 (4, 0)
  (0, 4)
   (4, 1)
    (1, 2)
     (2, 3)
      (3, 0)

In total we have a logarithmic number of outer iterations, each costs O(n log n), leading to the claimed complexity.

Implementation

def sort_class(s):
    """ sorts s and returns additional information

    :param s: string or list
    :returns p, c: p[j]=i if s[i] has rank j in sorted(s) and c[i] is rank of s[i] in sorted(set(s))
    :complexity: O(n log n) or better if sort makes use of specific values in s
    """
    S_index = [(x, i) for i, x in enumerate(s)]
    p = [i for x, i in sorted(S_index)]
    x2c = {x : i for i, x in enumerate(sorted(set(s)))}
    c = [x2c[x] for x in s]
    return p, c


def sort_cyclic_shifts(s):
    """ given a string s, sort lexicographically all cyclic shifts of s.

    The i-th cyclic shift of s is s[i:] + s[i:]
    :param s: string or list
    :returns L: such that L[j]=i if the i-th cyclic shift of s has rank j
    :complexity: O(n * log(n)^2)
    """
    p, c = sort_class(s)
    n = len(s)
    K = 1
    while K <= n:
        L = [(c[i], c[(i + K) % n]) for i in range(n)]
        p, c = sort_class(L)
        K <<= 1 
    return p

def suffix_array(s):
    """ given a string s, sort lexicographically suffixes of s
    :param s: string
    :returns: R with R[i] is j such that s[j:] has rank i
    :complexity: O(n log^2 n)
    """
    special = chr(0)
    assert special < min(s) 
    L = sort_cyclic_shifts(s + special)
    return L[1:]

Update September 2024

Programming is often a question of compromise. The implementation of sort_class is quite short. But its usage of a dictionary makes it a bit slow. The construction of array c can be improved by processing all items in order, as given by p, and keeping track of the distinct values seen so far. This gives an improvement of about 30% in a test we made. In many implementations the sorting of S_index is done by two stages of bucket sort. Since the keys are couples of ranks, we can sort first by the second rank, and then do a stable sort on the first rank. This will generate quite some lines of code in Python, and we don’t feel ready yet for so much compromise.

def sort_class(s):
    """ sorts s and returns additional information

    :param s: string or list
    :returns p, c: p[j]=i if s[i] has rank j in sorted(s) and c[i] is rank of s[i] in sorted(set(s))
    :complexity: O(n log n) or better if sort makes use of specific values in s
    """
    S_index = [(x, i) for i, x in enumerate(s)]
    p = [i for x, i in sorted(S_index)]
    c = [0] * len(s)
    curr_class = 0
    c[p[0]] = curr_class
    for i in range(1, len(s)):
        if s[p[i]] != s[p[i-1]]:
            curr_class += 1
        c[p[i]] = curr_class
    return p, c

References

an O(nlogn) implementation in CP-algorithms – describes applications
Visualization in action
Quest for the quickest implementation in Python

PC Trees

2024-01-03T00:00:00+00:00

A data structure representing all permutations satisfying constraints of the form: for a given set $S\subseteq\{0,1,\ldots,n-1\}$ the elements of the permutations on $\{0,1,\ldots,n-1\}$ have to be consecutive in circular manner.

Warning

We propose a simplified implementation which does not have the optimal time complexity. When restricting with a set $S$, the complexity won’t be in $O(|S|+p)$, where $p$ is the length of the terminal path (see below for definition), but in time $O(|S|+d)$, where $d$ is the total degree along the vertices of the terminal path.

To improve the implementation, one would need to replace the Python sets in P-nodes by a more sophisticated data structure.

However the performance still allows to solve the following problem in linear time in the size of the matrix. Given a matrix with 0,1 entries, the goal is to find, if possible, a permutation of its columns, such that in every row the 1’s are consecutive.

Acknowledgement

The little we know about PC-trees, we got it from the excellent ESA’2021 paper by Fink, Pfretzschner and Rutter.

Introduction

Suppose you want to shoot a film and need to decide in which order to film the different scenes, such that ideally every actor participates in consecutive scenes. Formally you are given a binary matrix $M\in\{0,1\}^{n\times m}$ and want to know if there is a permutation of its columns, such that in the resulting matrix every row matches the regular expression $0^\star1^\star0^\star$. Such a matrix is said to have the consecutive ones property (C1P for short). There exists a circular variant of this problem, where rows are in addition allowed to match the expression $1^\star0^\star1^\star$. In this variant, the column indices are considered to belong to the integers modulo $n$, namely $\mathbb Z_n$. These variants are essentially equivalent, since adding a column with only zeros reduces the non-circular variant to the circular one.

This problem has been introduced in 1899 by an archeologist named Petrie, who tried to place tombs on a timeline,assuming that ornaments observed in each tomb appeared at some moment in history and eventually became out of fashion. This situation could be modeled as the C1P-problem for a matrix $M$, where columns correspond to tombs, rows to ornaments, and a 1 in the matrix indicates the presence of an ornament in a particular tomb. In 1976 Booth and Luecker presented a data structure, called PQ-tree, which permits to solve the problem in linear time. Later in 1999 Shih and Hsu proposed a similar data structure called PC-tree, which was initially designed to embed planar graphs, and which solves the circular variant of the problem. According to Fink, Pfretzschner and Rutter, this data structure is easier to implement and more efficient.

Informal introduction

Given a permutation $\sigma$ on ${\mathbb Z}_n$ and a set $S \subseteq {\mathbb Z}_n$, we define the signature of $S$ in $\sigma$ as the binary string $b$, such that $b_i = 1$ if $\sigma_i \in S$ and $b_i=0$ otherwise. For example, for

         sigma = (7, 8, 3, 9, 4, 1, 2, 3, 0) 
         and S = {7, 8,       4, 1}
the signature is  1  1  0  0  1  1  0  0  0.

We say that $S$ is consecutive in $\sigma$ if its signature is of the form $0^\star1^\star0^\star | 1^\star0^\star1^\star$ (denoted as a regular expression). For example, for the above mentioned permutation $\sigma$, the set $\{1,4,7,8\}$ is not consecutive, but the set $\{0,2,3,7\}$ is consecutive and so is the set $\{1,4\}$.

For a given collection of sets $S_1,\ldots,S_k$ we want to maintain in a compact data-structure the set of all permutations for which each of the sets is consecutive.

For such a data structure, we want to be able to produce one of these valid permutations, and also to add a new set $S$ to the collection. The later operation will restrict the set of permutations stored in the data structure. We call this operation a restriction by $S$.

Other operations are possible, such as returning the number of valid permutations or choosing uniformly at random one of the valid permutations. But to keep this note simple, we do not consider these features.

This data structure will be represented by a tree, where the leafs are all the elements of ${\mathbb Z}_n$. Any depth first traversal of the tree will visit the leafs in some order. Inner nodes impose some restriction on the order to visit neighboring nodes, hence restricting the set of possible permutations.

Formal definition

A PC-tree is a tree consisting of $n$ leafs labeled from $0$ to $n-1$ and inner nodes of type P or C. C-nodes have a fixed order on their neighbors, while P-nodes have not. For simplification we assume that no inner node has degree 2.

Such a tree represents in compact form permutations on $\{0,1,\ldots,n-1\}$. These permutations can be obtained by a depth first traversal of the tree (DFS), starting from say the leaf labeled 0, and listing all leaf labels along the traversal. When exploring the neighbors of a P-node, we can explore them in arbitrary order. But when exploring the neighbors of a C-node, we can explore them either in clock-wise or counter-clock-wise order starting from the neighbor that lead to this C-node. These choices lead to all permutations encoded by the tree.

Example

The left tree represents all permutations on $\{0,1\ldots,9\}$. P-nodes are depicted as circles with simple border, while C-nodes have a doubled border, as in the right tree. The middle tree represents all permutations keeping the set $\{1,2,3,4\}$ consecutive. We say that the permutations are restricted by the set $\{1,2,3,4\}$. In the right tree we restricted also by the set $\{3,4,5,6\}$.

Restriction

The restriction of a PC-tree by a set $S$, transforms the tree such that it forbids permutations where elements of $S$ are not consecutive. The restriction is done by transforming the tree using the following steps, which are explained in the sequel of this note.

Test cardinality If the cardinality of $S$ is 0,1,n-1 or n, the restriction will not modify the tree, and the procedure ends here.
Label the nodes A leaf $i$ is full if $i\in S$ and empty otherwise. The labels propagate further to the inner nodes, which are labeled full, empty or partial. (full definition below)
Identify terminal path The smallest subtree $T$ spanning all the partial nodes is identified. If $T$ is not a path, then the restriction is not possible and the procedure is aborted. If $T$ is a path, we call it the terminal path.
Split nodes All vertices in $T$ are disconnected from each other. Partial nodes in $T$ are split into a full and an empty node.
Simplify The resulting list of nodes is modified as follows: Degree 1 nodes are replaced by their neighbor and C-nodes are replaced by their list of neighbors.
Reconnect nodes The list of nodes is connected to a new C-node.
Clean up The data structure needs to be cleaned such that all variables used for the restriction are again in their initial state. This action needs to be conducted also when the restriction procedure is aborted.

High level example

Suppose we have the following PC-tree.

Now we want to restrict it by the set $\{2,3,7,8,\ldots,14\}$. First we label the nodes as empty, partial or full. This is done starting from the leafs and propagating towards the inner nodes.

Then we identify the terminal path connecting all partial nodes.

Now we remove the edges between the nodes along the terminal path, resulting in a list of nodes (forming a forest). Partial nodes from this list are split such that the original nodes are connected to all empty neighbors, while the new nodes are connected to all full neighbors. This results in a circular list of nodes.

The list is simplified, in the sense that degree 1 nodes are replaced by their neighbor and C-nodes by their neighbor list.

A new C-node is created attaching all nodes from the list.

Finally the tree is cleaned, in the sense that all nodes are labeled empty as they were initially.

Structure of the implementation

We have 3 classes: Leaf, P_node and C_node which all inherit from a super-class Node. Nodes have

a pointer to a parent node, which encodes the orientation of the tree towards a root node. For the root node this attribute is None.
an identifier ID. This integer is used to represent the tree in text form.
an integer full_counter, which keeps track of the number of full neighbors.
inspectors is_full and is_partial, which determine the node label, according to the fullCounter`.
method signal_full used by a neighbor to signal to this node that it (the neighbor) became full.
method clean to reset the full_counter.
methods to attach and detach nodes with each other. These methods maintain the parent pointer. detach_bilateral removes an edge between two nodes, by detaching them on both endpoints. attach_neighbors attaches a node to all its neighbors.

A PC_tree is a class which has

an initializer with a given number n
a method restrict which restricts the tree by a given set $S\subseteq \{0,1,\ldots, n-1\}$
a method represent which returns a canonical representation of the tree in form of a list of list. Each inner list represents an inner node, and contains a character P or C for the type, its identifier and its list of neighbors in lexicographical minimal order. This list can optionally end with the identifier of the parent node.
a method frontier which returns a valid permutation represented by the tree. With some little work it is possible to extend this method to make it select uniformly at random a valid permutation, or to return the lexicographically smallest one.

These 3 subclasses have in common

a method to_signal which for a full node returns the neighbor to which a signal needs to be send. For a leaf it is just the inner node to which the leaf is attached to, and for inner nodes it is the unique neighbor which hasn’t be signaled.

P and C-nodes have in common

an inspector is_splittable which tests if the node is splittable, which means roughly that its full neighbors are or can be made adjacent.
a method split which returns a new node with only the full neighbors.
an attribute neighbors, which is a sequence for the C-node and a set for the P-node. In addition C-node has an attribute first_full which is the first full neighbor to have signaled to this node. It is used as a starting point to explore the neighbors in order to find a maximal interval of full neighbors.

C-nodes have a method flip, which inverts the order of the neighbors.

In addition we have an exception called Infeasible which is raised whenever we find out that the restriction fails.

Labeling

Inner nodes are labeled as follows. Initially all inner nodes are empty. When all but one neighbor of a node become full, then the node becomes also full. This process is implemented by a signaling procedure using a queue, starting with the full leaves. Every node in the queue signals to its unique non-full neighbor that it became full, and is removed from the queue. And when this neighbor has enough full neighbors, then it becomes full and joins the queue.

Identifying the terminal path

This is trickiest part of the procedure. We would like the complexity to be linear in the maximum distance between partial nodes. So a tree traversal to compute the distances is too costly. In principle the tree is not rooted, but if we maintain an orientation towards a root, then we can obtain the aimed time complexity.

Formal problem

We are given a rooted tree on a vertex set $V$. There is a function $f : V \mapsto V \cup \{ \bot \}$, which returns the ancestor for every non-root vertex, and returns $\bot$ for the root vertex. In addition we are given a set of vertices $S$, which we call the seeds. The goal is to decide if there is a simple path $P$ containing all vertices $S$. Here simple means that $P$ contains no cycles.

Let $a$ be the lowest common ancestor of all vertices in $S$, which is also called the apex. Denote by $S \rightarrow a$ the union of all paths connecting $a$ with every vertex in $S$. So the problem consists in deciding if $S \rightarrow a$ is a path or not. We aim for an algorithm with complexity $O (| S \rightarrow a |)$.

For a yes-instance, the resulting path is completely described by the apex $a$ and one or two vertices in $S$, which are the extreme points of the path and called tails.

In the particular case when $S$ is a singleton set, we can answer yes, because this single vertex forms itself the resulting path. So from now assume $| S | \geqslant 2$.

The rough idea of the algorithm is quite simple. We walk up in the tree from every vertex in $S$, and do this in parallel at constant speed. In fact we walk up one step at a time in round robin among the seeds. Hence from every seed we grow a path towards the root. Eventually the resulting paths will run into each other. In case of such a collision, the path which ran into another path stops growing, and we say that its seed becomes inactive. When a path reaches the root, it becomes inactive as well.

If we would let this procedure run until all vertices are inactive, then the union of the paths contains a subtree of the original tree. When we use the word in-degree, it is meant with respect to this subtree. All its leafs are seeds in $S$, however not every seed is a leaf. One of the seeds would become inactive because it reached the root (or tries to make one additional step from the root). This seed is called the leader. If the union of those paths contains a vertex of in-degree higher than 1, then it must be the unique such vertex and its in-degree must be 2. This vertex must be the apex and there must be exactly 2 tails. We say that the resulting path has an A-shape. Otherwise if the union is a single path all the way up to the root, then there is a unique tail, and the highest seed is the apex. We say that the resulting path has an I-shape.

Our objective is to detect these two shapes, and to avoid that the top-most seed path grows unnecessarily towards the root, which could exceed the requested time complexity. Doing this carefully is quite subtle.

The algorithm

When growing a path from seed $v$, we mark all vertices of the path by $v$, including the seed itself. This allows us to detect when one path runs into another.

A seed can be either active or inactive. Initially all seeds are active. Also we maintain a set $T \subseteq S$ of all vertices which could potentially be the final tails. In addition we have a variable $a$ storing a vertex which potentially could be the apex. And finally we have a variable $\ell$, containing the leader seed, once we know it. Initially $a$ and $\ell$ are empty (denoted $\bot$) and $T = S$.

event	#active	$\|T\|$	$a$	$\ell$
initial state	$= n$	$= n$	$= \bot$	$= \bot$
attempt to leave root	-1			$\neq \bot$
run into terminal seed	-1	-1
run into non-terminal seed	-1		$\neq \bot$
run into marked non-seed	-1		$\neq \bot$
final state 1 (I-shape)	=0	=1	$= \bot$	$\neq \bot$
final state 2 (I-shape)	=1	=1	$= \bot$	$= \bot$
final state 3 (A-shape)	=0	=2	$\neq \bot$	$\neq \bot$
final state 4 (A-shape)	=1	=2	$\neq \bot$	$= \bot$

Table 1: Changes of the variables triggered by various events. A $\neq \bot$ in the last two columns means that one of the variables $a, \ell$ receives a vertex.

The path growing from active seeds will be extended in Round-Robin manner. During the updates some active seeds might become inactive. Consider a path emerging from seed $v$ and leading to some vertex $p$. Let $q = f (p)$ be its ancestor. We will extend the path by the edge $(p, q)$ and conduct the following actions.

If $q = \bot$, then $p$ was the root, and $v$ becomes inactive. Also we set $\ell :=v$.
If $q$ is marked, then $v$ becomes inactive. The in-degree of $q$ increases. Instead of explicitly storing the in-degrees we can make use of the set $T$ and the variable $a$, as follows.
- If $q \in S \cap T$, then $q$ has in-degree 1, and we remove $q$ from $T$, as $q$ is definitely not a tail.
- If $q \not\in T$ (i.e. $q \in S \setminus T$ or $q \not\in S$), then $q$ definitely has in-degree at least 2. In this case $q$ is potentially the apex. So we set $a = q$ if $a$ was empty. However if $a$ was not empty, then we detected a situation which should not happen and we can abort the algorithm, reporting that the restriction failed.
In all other cases we mark $q$ by $v$.

All active seeds will be extended in Round Robin manner until the number of active seeds plus the number of leaders (0 or 1) becomes 1. In other words, the procedure ends either when there is no leader and a single active seed or there is a leader and no active vertex.

At this point, if there is no leader, we set the leader $\ell$ to be the unique active seed. Two cases are to be distinguished.

If there is an apex, i.e. $a \neq \bot$, then the resulting path is in an A-shape, and we check that $\ell$ is the mark at $a$. If yes, we can safely report that this is a yes-instance (meaning that the restriction succeeded). Otherwise, we know that $\ell$ is above $a$, vertex $a$ would have degree 3, and we report that this is a no-instance (meaning that the restriction failed).

In case there is no apex, then the resulting path is in an I-shape. The apex is the leader.

Implementation details

For the marked vertices, we use a set marked storing id’s of marked vertices. For the set $T$ we either use a set called terminal, or a boolean attribute of a node, together with a counter nbTerminal of terminal nodes. For active vertices, we use a dictionary active, mapping vertex id’s to the current endpoint of the path. The tree defining function $f$ is in fact realized by a node attribute parent. The variable $a$ is called apex, and the variable $\ell$ is called leader.

In case of a yes-instance, we return the list of the vertices of the terminal path. And in case of a no-instance we raise an exception.

Splitting

Suppose that we already identified the terminal path $T$. Now we need to split the nodes along $T$ so that full nodes become adjacent and empty nodes become adjacent as well. Before modifying our data structure, we need to make sure that the nodes along the path $T$ can be split. This is done by a dry run, checking individually each node on $T$. P-nodes can always be split, so we focus on C-nodes.

Verifying

Formally we have the situation of a C-node, which might have a left and right neighbor in $T$ (except for the first and last node on $T$). So we have variables left_terminal and right_terminal, which each contain a node or are None. We also know one full neighbor, which is stored in an attribute called first_full, and the number of full neighbors stored in an attribute called full_counter.

By expanding to the left and to the right from first_full, we reach an interval of full nodes spanning from left to right. First we must verify that all full neighbors are in this interval, otherwise the node is not splittable.

Let $x$ be the left neighbor of left and $y$ be the right neighbor of right. We must verify that $x$ is the given left_terminal (unless it is None). Similarly we must verify that $y$ is the given right_terminal (unless it is None). It might be necessary to flip the C-node for this purpose.

Actual splitting

First we need to remove the connections between the nodes along the terminal path $T$. Then for each node in $T$ we detach all the full neighbors and attach them to a new node (of the same type P or C). This results in a circular node list $L$.

Then we simplify this list, in the following sense. For every node $x$ in $L$, which has a single neighbor $y$, we replace $x$ by $y$ in $L$. This avoids creating degree 2 nodes.

Finally we create a new C-node with $L$ as its neighborhood.

A generalization to partially defined matrices

We consider a generalization of the C1P-problem, where we are given a matrix $M\in\{0,1,\star\}^{n\times m}$ with the interpretation that a $\star$ in $M$ could be either 0 or 1. Now the question is to decide if it is possible to replace all these wildcards by 0 or 1, such that the resulting matrix has the consecutive ones property. In this section we do not consider the circular variant of the C1P property.

The C1P-problem with wildcards is NP-complete.

Proof: Clearly the problem is in NP, since one can verify in polynomial time if for a given permutation of the columns all rows match the regular expression $[0\star]^\star [1\star]^\star [0\star]^\star$. Matching this expression is a sufficient and necessary condition for the existence of a replacement of the wildcards resulting in a row where all ones are consecutive, i.e which matches $0^\star1^\star0^\star$.

To show NP-hardness we reduce from Betweenness, which is the following NP-complete problem. Given a positive integer $n$ and a sequence of $k$ triplets in $\{1,\ldots,n\}^3$, decide if there is a permutation $\sigma$ on $\{1,\ldots,n\}$ such that for every given triplet $(a,b,c)$, $\sigma_b$ is between $\sigma_a$ and $\sigma_c$, i.e. $\min\{\sigma_a,\sigma_c\} < \sigma_b < \max\{\sigma_a,\sigma_c\}$. Each instance of Betweenness $R$ is mapped to an instance $M$ of the C1P-problem with wildcards, such that $M$ consists of $n$ columns and $m=2k$ rows. Every triplet $(a,b,c)\in I$ is mapped to distinct two rows in $M$. The first row has $1$ in columns $a,b$ and $0$ in column $c$, while the second row has $1$ in columns $b,c$ and $0$ in column $a$. The remaining entries of the rows are $\star$.

\[\begin{array}{ccccccc} &a&&b&&c&\\\hline \star\ldots\star & 1 & \star\ldots\star & 1 & \star\ldots\star & 0 & \star\ldots\star \\ \star\ldots\star & 0 & \star\ldots\star & 1 & \star\ldots\star & 1 & \star\ldots\star \end{array}\]

In any valid permutation $\sigma$, one of $\sigma_a,\sigma_b,\sigma_c$ is between the other two. The first row forbids $\sigma_c$ to be between the other two and the second row imposes the same constraint on $\sigma_a$. As a result only $\sigma_b$ can be between the other two values. This observation is enough to ensure that every solution to the instance $I$ of Betweenness is also a solution to the instance $M$ of C1P with wildcards and vice-versa. This concludes the proof.

Conclusion

It is not easy to implement a PC-tree data structure. If we had to redo the project, maybe we would have chosen C++ instead of Python. Since Python is not suited for long codes like this one.

References

David G. Kendall. Incidence matrices, Interval graphs and Seriation in Archaeology. Pacific Journal of Mathematics, 1969.
Kellogg S. Booth and George S. Lueker. Testing for the Consecutive Ones Property, Interval Graphs and Graph Planarity Using PQ-Tree Algorithms. Journal of Computer System Science, 1976.
Wei-Kuan Shih and Wen-Lian Hsu. A new planarity test. Theoretical Computer Science, 223(1-2):179–191, 1999.
Murray Patterson. Variants of the Consecutive-Ones Property Motivated by the Reconstruction of Ancestral Species, Doctoral dissertation, University of British Columbia, 2012.
Simon D. Fink, Matthias Pfretzschner, Ignaz Rutter. Experimental Comparison of PC-Trees and PQ-Trees, In 29th Annual European Symposium on Algorithms (ESA 2021) (Vol. 204, p. 43), 2021.
The above described implementation.

Largest rectangle under an histogram

2023-12-22T00:00:00+00:00

You are given an histogram and want to identify the area of the largest rectangle that fits under the histogram.

There is a solution in linear time using a stack. There is also a divide-and-conquer solution that we describe here.

On range $[l, r]$:

Find the minimum $m$
One rectangle candidate is $m \times (r - l + 1)$
Other rectangle candidates are recursively computed on the left and right of the minimum.

So the complexity overall verifies: $T(n) = 2T(\frac{n}2) + f(n)$ where $f(n)$ is the cost of finding the minimum over an interval of length $n$.

Finding the minimum efficiently over a range

This task can be done:

Either in $O(n)$ naively
In $O(\log n)$ using a range minimum query structure
Or even in $O(1)$ using a sparse table.

Overall complexity

If $f(n) = O(n)$, master theorem says complexity is $T(n) = \Theta(n \log n)$.
If $f(n) = O(1)$, master theorem says complexity is $T(n) = \Theta(n)$.
If $f(n) = O(\log n)$, well master theorem can’t be used; we can use instead a generalization called the Akra-Bazzi theorem (1998) with $p = 1, g(x) = \log x$:

\[\begin{align} T(x) & = \Theta\left(x\left(1 + \int_1^x \frac{g(u)}{u^2} du\right)\right)\\ & = \Theta\left(x\left(1 + \left[- \frac{\log x + 1}{x} + 1\right]\right)\right)\\ & = \Theta(2x - \log(x)) & = O(x). \end{align}\]

AI and LLM in education

2023-12-05T00:00:00+00:00

In my new role as scientific advisor at the French Ministry of Education, I have recently been looking for meta-analyses and RCTs about impact of AI (and LLMs) in education.

A recent (preprint, Dec 1) RCT by University of Toronto and Microsoft Research on 1200 participants.
UK Dept of Education report about LLMs in Education
Meta analysis about ITS

A recent (preprint, Dec 1) RCT by University of Toronto and Microsoft Research on 1200 participants.

People were recruited on Amazon Mechanical Turk and were shown SAT-like math questions.

RQ1. When doing practice questions for a math test, how does the type of explanation people receive (answers alone or answers with LLM-generated explanations) affect performance on subsequent test questions?
RQ2. How does the relationship between explanation type and performance change when people i) attempt questions before seeing explanations or ii) see explanations before attempting questions?

LLM-based explanations positively impact learning (relatively to seeing only correct answers), regardless of whether participants consulted them before or after attempting practice problems.

An accompanying qualitative analysis revealed that these boosts in performance were indeed due to participants adopting the strategies they were shown, and that exposure to LLM explanations increased the amount people felt they learned and decreased the perceived difficulty of the test problems.

Kumar, Harsh, et al. “Math Education with Large Language Models: Peril or Promise?.” Available at SSRN 4641653 (2023). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4641653

UK Dept of Education report about LLMs in Education

From June 14 to August 23, the English government asked teachers how they were using LLMs for education. Last week, they published a very interesting report summarizing the 567 responses they obtained.

More than 2/3 respondants reported a positive result. Other 1/3 reported mixed or negative impact or noted that it was too early to tell.

Access to GenAI is not a subtitute for having a deep reservoir of subject knowledge held in your long-term memory.

GenAI’s ability to quickly generate explanations at varying educational levels could help explain complex concepts to pupils in a way they understand. Struggling students could be quickly identified and supported, while high-performing students could be challenged with more advanced materials.

Opportunities

freeing up teacher time for focusing on teaching
- for creating and improving educational resources
- streamlining administrative tasks (emails, summarising meeting minutes, repetitive tasks)
- automating marking and assessment (experimental)
- generating (regular) feedback on students’ work; also, personalized study and revision plans for pupils based on their performance
- lesson and curriculum planning
- live demos (e.g. in food and nutrition lessons; or acting as a scriptwriter for the drama department, ChatGPT has sparked creativity)
providing (adaptive) additional educational support (notably for students with special needs)
- enhanced engagement
- improved accessibility and inclusion
proofread, edit and improve written content (first draft)
support coding
teacher professional development: understanding the latest pedagogical strategies (one respondent suggested creating personalized learning pathways for teachers based on their skills)
(provided there is an improved access to technology and the Internet)

Risks and concerns

over-reliance on GenAI tools (among pupils)
young people should not access or create harmful or inappropriate content (academic misconduct)
pupils should understand their personal data is being processed using AI tools
ensuring pupils’ work is their own
GenAI tools can produce unreliable or biased information: any content produced requires professional judgement
(exacerbating the “digital divide”, as some pupils do not have access to devices, or stable Internet)

Negative impact

Some teachers reported suspected academic malpractice at their institution
Teachers needed additional time to check signs of AI use
Some GenAI outputs were of poor quality, which then took time to correct to a sufficient standard for use

Challenges

User knowledge and skills (e.g. awareness of potential applications, or prompt engineering)
Performance of tools (inaccurate and biased content, “issues such as Americanized spelling” 😂)
Workplace lack of awareness and skeptical attitudes or fear (some institutions have blocked GenAI tools on devices)
Data protection adherence: non-compliance with GDPR
Managing student use: exposure to harmful content
Access: paywalls to premium versions, institutions banning platforms, accessibility of student with special needs

Meta analysis about ITS

Cover trees

2023-11-02T00:00:00+00:00

Cover a tree with paths or caterpillars.

Cover with descending paths

Fix an arbitrary root in the given tree. Now for every vertex $v$ we have the notion of its subtree $T_v$ rooted at $v$.

We say that a path is descending, if for one of its extreme points $v$, the path is contained in $T_v$.

If we want to cover a tree with descending paths, then this is quite easy. For example there is the trivial solution consisting for every vertex $v$ of the singleton vertex path ${v}$. Alternatively we could choose for every inner vertex $v$, an arbitrary descendent $u$. The set of selected edges $(u,v)$ would form again a solution.

So usually we impose some additional conditions.

Heavy-light decomposition

The original definition is slightly different, but this one is easier to implement, and has the same desired property.

Here we want to choose for every inner vertex $v$, a descendent $u$ which maximizes $|T_u|$. Ties can be broken arbitrarily. These edges are called heavy. All other edges are called light. Now we have the property, that for every pair of vertices $u,v$ with lowest common ancestor $a$, both paths $u-a$ and $a-v$ traverse at most $\log n$ heavy paths. This is because every light edge $(u,v)$ — with $u$ descendant of $v$ — has the property that $T_u$ is at most half as big as $T_v$. Hence each of the two paths can contain at most a logarithmic number of light edges.

This is interesting when we have to maintain a datastructure on the tree, such vertices are labeled with numbers, and we want to add a value $k$ to every vertex along the path between two given vertices $u,v$, or return the sum of these labels along the path. If the tree were a line graph, we could use a segment tree for this purpose. Here we use a segment tree for each heavy path. In fact we can use one big segment tree, with portions of it corresponding to heavy paths.

Here is an excellent detailed explanation.

Maximizing total path length

Here we want to remove a minimum number of edges in the tree, such that the result consists of a collection of paths. In other words, for every vertex $v$ of degree $d$ at least $3$, we have to remove $d-2$ adjacent edges. Call this number the deletion number of vertex $v$.

When removing an edge, it could be between to vertex of positive deletion number, or adjacent only to one. Ideally we prefer the first type of edges. But we cannot remove them greedily, see the example below.

      (a) (b) (c)
*---1---1---1---1---*
    |   |   |   |
    *   *   *   *

If we remove the middle edge (b), then we need to remove 2 addition edges. However the optimum here is to remove edges (a) and (c).

We can solve the problem greedily using a different approach. For every vertex $v$, let $A_v$ be the maximum number of edges in a covering of $T_v$ with paths. Let $B_v$ be the same number but with the restriction, that $v$ is the extreme point of a path. This includes the case when $v$ is covered by a singleton vertex path.

Now we say that vertex $u$ is interesting if $A_u=B_u$.

For a leaf $v$ we have $A_v=B_v=0$, and $v$ is interesting.

Consider a vertex $v$ and all its descendants $u$.

If none of the descendants is interesting, then $A_v=B_v=\sum A_u$, and $v$ is interesting.
If a single descendant $u_0$ is interesting, then $A_v=B_v=1+\sum A_u$, and again $v$ is interesting.
If there are at least two interesting descendants, then $A_v=2+\sum A_u$, $B_v=1+\sum B_u$, and $v$ is not interesting.

We observe that $B_v$ is either $A_v$ or $A_v-1$.

Cover a tree with caterpillars

A caterpillar is a tree, which consists of a single path, with leafs attached to it. In other words, every vertex of degree at least $3$, can have at most 2 non-leaf neighbors. The goal is to cover the tree with caterpillars, maximizing the total number of edges in these caterpillars.

The optimal covering with caterpillars can be computed with dynamic programming. But we could only find a tedious solution. Which we sketch here.

/*
  Airbus vs. Boeing
  http://uva.onlinejudge.org/contests/258-febc44a2/11862.html

  dynamic programming

  We want to remove a minimal number of edges to turn a forest in a
  graph that can be drawn without crossings as a bipartite
  graph. These graphs are called caterpillars. A caterpillar is a tree
  where the longest path has only leafs attached to it.

  our notation for caterpillars:

  a-node : has two neighbors of degree>1
  b-node : has degree>1 and one neighbor of degree>1
  c-node : has degree=1

  c--b--a--a--a--b--c
    / \    |    /|\
   c   c   c   c c c

  DFS fixes an orientation of the trees.  
  Tv := subtree with root v
  p[v] = parent node of v in the tree

  in the following asum, bsum, csum refers to the sum over sons of v
  and k to the number of sons
  the type of the father is without taking into accounts Tv

  D[v] = max where v has no father to connect to
  
  C[v] = max where v has a degree 0 father
  
  B[b] = max where v has a type B father

  A[v] = max where v has a type A father, 
         (or a B father, but v can only be a leaf of its father)

  let CD0 be the maximum of c[u]-d[u] among all sons u of v
  let BA0 be the same for b[u]-a[u]
  let BA1 be the second maximum value of this difference

  then

  D[v] = 0 of v has no son, otherwise
       = max{ dsum,     -- do not connect to sons
              dsum+CD0, -- become leaf of a single son
              asum,     -- become a star
              asum+BA0, -- let one son to become B
              asum+BA1} -- let two sons become B

  C[v] = max{ D[v],       -- don't connect to father, otherwise...
              1+asum,     -- become star
              1+asum+BA0, -- become B
              1+asum+BA0+BA1 }  -- become A with two B sons

  B[v] = max{ D[v],       -- don't connect to father, otherwise...
              1+asum,     -- become B (father becomes A)
              1+asum+BA0 } -- become A with one B son

  A[v] = max{ D[v],       -- don't connect to father
              1+dsum }    -- become leaf of its father

  Complexity is O(N log N) but could be made linear.
*/

#include <iostream>
#include <vector>
#include <map>
#include <algorithm>

using namespace std;

const int MAX=200;

int    n; // number of vertices

map<string,int> name;

// internally vertices have indices
int name2idx(string s) {
  if (name.find(s) == name.end()) {
    int i = name.size();  // assign new number
    name[s] = i;
  }
  return name[s];
}

int d[MAX];        // degrees
int E[MAX][MAX];   // neighbors : N[u][i] = i-th neighbor of u
int p[MAX];        // parent node in tree, -1 for root, -2 not yet visited
const int ROOT=-1, NOT_YET_VISITED = -2;
int A[MAX], B[MAX], C[MAX], D[MAX];

#define forallNeighbors(v,u) for(int u,i##u=0; (u=E[v][i##u], i##u<d[v]); i##u++)
#define forallSons(v,u)      forallNeighbors(v,u) if (p[u]==v)
#define forallVertices(v)    for(int v=0; v<n; v++)

void DFS(int f, int v) {
  p[v] = f;
  forallNeighbors(v,u) 
    if (p[u] == NOT_YET_VISITED)
      DFS(v,u);
  //                              c
  int asum = 0, dsum = 0;
  int k    = 0;  // number of sons
  vector<int> BA, CD;

  forallSons(v,u) {
    asum += A[u];
    dsum += D[u];
    k++;
    BA.push_back(B[u] - A[u]);
    CD.push_back(C[u] - D[u]);
  }
  sort(   BA.begin(), BA.end());
  reverse(BA.begin(), BA.end());
  sort(   CD.begin(), CD.end());
  reverse(CD.begin(), CD.end());

  int BA0 = k>0 && BA[0]>0 ? BA[0] : 0;
  int BA1 = k>1 && BA[1]>0 ? BA[1] : 0;
  int CD0 = k>0 && CD[0]>0 ? CD[0] : 0;
  
  //                              d
  if (k==0)
    D[v] = 0;
  else 
    D[v] = max( dsum+CD0, asum+BA0+BA1 );
  //                              c
  C[v] = max(D[v], 1+asum+BA0+BA1 );
  //                              b
  B[v] = max(D[v], 1+asum+BA0 );
  //                              a
  A[v] = max(D[v], 1+dsum );
}

int DFS() {
  int total=0;
  forallVertices(v)
    p[v] = NOT_YET_VISITED;   // use parent label to mark non-visited
  forallVertices(v)
    if (p[v]==NOT_YET_VISITED) {
      DFS(ROOT, v);
      total += D[v];
    }
  return total;
}

int main() {
  for (int testCase=1; true; testCase++) {
    cin >> n;
    if (n<=0) break;
    if (n>MAX) for (;;) {}
    name.clear();
    //                            read vertices
    forallVertices(v) {
      string vertex;
      cin >> vertex;
      d[name2idx(vertex)] = 0;
    }
    //                            read edges
    int m;
    cin >> m;
    string nameU, nameV;
    while (m-->0) {
      cin >> nameU >> nameV;
      int u = name2idx(nameU);
      int v = name2idx(nameV);
      E[u][d[u]++] = v;
      E[v][d[v]++] = u;
    }
    cout << "Case " << testCase << ": " << DFS() << endl;
  }
  return 0;
}

Meilleur developpeur de France 2023

2023-03-11T00:00:00+00:00

Le 9 mars 2023 avait lieu la compétition Meilleur Développeur de France 2023. Les algorithmes nécessaires sont résoudre les problèmes sont assez simples, mais il faut coder rapidement. Après coup, et à tête reposée, voici comment on aurait pu résoudre certains des problèmes.

Retrouvez les problèmes ici.

MDF round 1 Pizza - Découpage des pizzas

Déterminer le nombre de composantes connexes. Pour chaque cellule $(i,j)$, si elle contient ‘#’, alors augmenter un compteur et composantes et explorer la composante par un parcours BFS ou DFS, comme vous préférez. Utilisez la même matrice, pour marquer les cellules visitées, par exemple par la lettre M.

MDF round 2 Jeux Olympiques - Ascenseurs

On vous donne $m$ intervalles de la forme $[\ell_i, r_i]$ et on veut savoir si leur union inclut un intervalle donnée $[e,n]$.

Fausse piste: Si vous construisez un graphe, où chaque intervalle $[\ell_i, r_i]$, génère une arête $(\ell_i,r_i)$, alors il vous manque des informations dans votre modélisation. L’ascenseur peut servir toutes les stations entre $\ell_i$ et $r_i$, pas seulement relier les extrémités.

Difficulté. Les extrémités des intervalles sont entiers. Et on doit distinguer une instance $[1,1],[2,2]$ de l’instance $[1,2]$, qui elle peut relier les étages 1 et 2.

Technique: on va balayer les intervalles de gauche à droite. Soit $T$ l’ensemble de toutes les extrémités d’intervalles plus les valeurs $e,n$. Pour chaque $\textrm{etage}\in T$ en ordre croissant, on maintient une valeur couvert, qui dit combien d’ascenseurs peuvent aller de cet étage à l’étage suivant. Un compteur C[etage] indique de combien cette valeur change, d’un étage au suivant dans $T$.

import sys
from collections import Counter

def readints(): return list(map(int, readstr().split()))

n, m, e = readints()
A = [readints() for _ in range(m)]


def solve(A, low, high):
    C = Counter({low:0, high:0})
    for left, right in A:
        if left < right:
            C[left] += 1
            C[right] -= 1
    L = [(etage, C[etage]) for etage in C]
    # print(L)
    couvert = 0
    for etage, delta in sorted(L):
        couvert += delta
        if not couvert and low <= etage < high:
            return False
    return True 

if solve(A, e, n):
    print("YES")
else:
    print("NO")

MDF round 3 Basket - Égalité au tableau d’affichage

D’abord on détermine manuellement les couples de chiffres qui s’obtiennent par exactement un changement sur l’affichage. Puis on essaye les trois possibilités de corriger le premier, deuxième ou troisième chiffre.

import sys

def readints(): return list(map(int, readstr().split()))


couples = [ 
[0, 8],
[5, 6],
[6, 8],
[1, 7],
[3, 9],
[5, 9]]

possible = set()
for a, b in couples:
    possible.add((a, b))
    possible.add((b, a))

def solve(L):
    a, b, c = L 
    if (a, c - b) in possible:
        return (c - b, b, c) 
    if (b, c - a) in possible:
        return (a, c - a, c)
    if (a + b, c) in possible:
        return (a, b, a + b)
    return None 
    

abc = solve(readints())
if abc:
    print(*abc)
else:
    print("Impossible")

MDF round 5 Poubelles - Le dédale du local

Étant donnée une fonction $f$ de ${1,\ldots,n}$ dans ${1,\ldots,n}$, et un entier $k$ calculez $f^k(1)$, où $f^k$ est la fonction $f$ itérée $k$ fois. Le problème est que $k$ est énorme, 10 puissance 12. Plusieurs solutions sont possibles.

Solution en $O(n \log k)$ par exponentiation rapide

Étant donnée $f$ on peut calculer en temps $O(n)$, la fonction $f^2$. Puis avec la même méthode, étant donnée $f^2$ on peut calculer en temps $O(n)$, la fonction $f^4$, et ainsi de suite, jusqu’à $f^{2^{40}}$. Puis on peut décomposer $k$ en somme de puissances de 2, et appliquer les fonctions correspondantes. Par exemple, pour $k=13=8+4+1$, on aurait $f^k(1) = f^8(f^4(f^1(1)))$.

Solution en $O(n)$ par détection de cycle

La fonction $f$ itéré sur la valeur initiale $1$, produit une séquence infinie $(a_i)_{i\geq 0}$ avec $a_0=1$, qui se répète au bout d’un moment. Il existe alors deux entiers $p,q$, tel que pour $k \geq p$, on ait $f^k(1) = f^{p + ((k-p) \bmod q)}(1)$. On peut trouver ces deux entiers en construisant la séquence $(a_i)_i$ et en associant dans un dictionnaire pour chaque $v$ de la séquence, le premier indice $i$ avec $a_i=v$. Une technique alternative est celle du lièvre et de la tortue, voir ici et ici.

MDF round 5 Chocolat - Vox populi

Notre solution a une complexité de $O((n + k )2^k)$, où $k=13$ est le nombre maximum d’ingrédients au total. L’idée est d’abord de remplacer les ingrédients par un numéro unique 0,1,..,12. Puis chaque client $i$ correspond à un ensemble $C_i$ de 3 entiers. Un ensemble $S$ est une solution si et seulement si il intersecte tous les ensembles $C_i$.

Pour comprendre ce code, il faut connaître la technique de codage des ensembles dans les entiers. Le singleton {i} est représenté par 2 puissance i, qui s’écrit 1 << i. L’intersection se calcule avec le ET binaire, qui s’écrit &. La différence symétrique, s’écrit ^, l’union | et pour un ensemble non-vide $S$ dont le minimum est $i$, l’ensemble singleton {i} s’écrit S & -S.

import sys

def readint(): return int(sys.stdin.readline())
def readstr(): return sys.stdin.readline().strip()
def readints(): return list(map(int, readstr().split()))

# lire l'entrée

n = readint()
client_str = [readstr().split() for i in range(n)]

# associer à chaque ingrédient un numéro unique 0, 1, 2, etc
# set permet d'enlever les doublons
# list permet de leur associer des indices

L = list(set(ingr for c in client_str for ingr in c))

# rank[s] est le numéro de l'ingrédient s (de type str) 

rank = {s:i for i, s in enumerate(L)}

# maintenant chaque client correspond à un ensemble de 3 ingrédients
# on code les ensembles dans un bitvector

client = [0] * n 
for i, ci in enumerate(client_str):
    for ingr in ci:
        client[i] |= 1 << rank[ingr]

# comme il n'y a que 2**13 candidats pour les solutions, on les essaye tous

def cardinality(S):
    retval = 0
    while S:
        retval += 1
        S ^= S & -S 
    return retval 

def is_solution(S):
    for required in client:
        if S & required == 0:
            return False 
    return True
    
best = 14 # = infini pour ce problème

m = len(rank)
# essayer toutes les solutions (sauf l'ensemble vide)
for S in range(1, 1 << m):
    if is_solution(S):
        c = cardinality(S)
        if c < best:
            best = c 
print(best)

MDF finale Classement à la main

On vous donne une liste de paires $(t, s)$, où $t$ est un nombre et $s$ une chaîne de caractères. On stocke dans un dictionnaire $C[s]$ le nombre de paires avec la chaîne $s$. Puis on crée une liste $L$ avec toutes les valeurs $t$, des paires $(t,s)$ pour lesquelles $C[s]=1$. Il faut afficher cette liste dans l’ordre triée.

MDF finale Tricheurs

On vous donne un graphe avec $n=10.000$ sommets et $m=20.000$ arêtes. Et on vous donne également un ensemble de sommets $T$, et un sommet particulier $v_0\not\in T$. Et on veut le nombre de sommets, qui soient strictement plus proche de $v_0$ que de tout sommet dans $T$.

L’idée clé est de contracter $T$ en un seul sommet $v_1$. Puis il suffit de faire deux parcours en largeur pour calculer les distances depuis $v_1$ et depuis $v_0$ pour au final comparer les distances sommet par sommet.

MDF finale Meilleure startup de France

On vous donne une matrice $T$ avec 2 lignes et $n$ colonnes. Les entrées sont des textes. Puis on demande de calculer le nombre de vecteurs binaires $b$ de longueur $b$, tel que pour tout $i$, $T[b[i]][i]$ soit different de $T[b[i-1]][i-1]$, où il faut comprendre le $i-1$ modulo $n$. Appelons un tel vecteur valide. Une exception est le cas $n=1$, où il faut juste répondre $2$.

Nous modélisons le problème comme suit. Si on restreint au $k$ premières colonnes de $T$, on a un problème plus petit. Nous calculons une matrice $A^k$ de dimension $2\times 2$, tel que $A^k[u][v]$ est le nombre de vecteurs $b$ valides pour ce problème restreint avec $b_1 = u$ et $b_k=v$.

Pour le cas de base, $A^0$ est la matrice d’identité.

Pour le cas $n>1$, $A^n$ se calcule par le produit $A^{n-1}$ avec une matrice $B$, qui code la compatibilité des choix pour $b_{n-1}$ et $b_n$. Concrètement $B[u][v]$ est 1 si $T[u][n-1]\neq T[v][n]$ et 0 sinon.

La réponse au problème est la somme $A^n[0][0] + A^n[1][1]$. Ceci prend compte du fait que la dernière colonne de $T$ est également la colonne qui est le prédécesseur de la première colonne de $T$ dans l’ordre circulaire.

#!/usr/bin/env pypy3

""" c.durr - 2023 - mdf

"""


import sys

def readint(): return int(sys.stdin.readline())
def readstr(): return sys.stdin.readline().strip()
def readints(): return list(map(int, readstr().split()))

def matrix(before, after):      # compatibilité
    B = [[0,0], [0,0]]
    for i in (0,1):
        for j in (0,1):
            B[i][j] = int(before[i] != after[j])
    return B

def mult(A, B):                 # multiplication de matrices
    C = [[0,0], [0,0]]
    for i in (0,1):
        for j in (0,1):
            for k in (0,1):
                C[i][j] += A[i][k] * B[k][j]
    return C 
    
def solve(L):
    if len(L) == 1:
        return 2
    A = [[1,0],[0,1]]           # identité
    for i in range(n):
        B = matrix(L[i - 1], L[i])      
        A = mult(A, B)
    return A[0][0] + A[1][1]

n = readint()
L = [readstr().split() for _ in range(n)]
print(solve(L))

SWERC 2022 Practice Session - Bloggers

2023-02-24T00:00:00+00:00

Abstract: Maintain two tables $t_0$, $t_1$. Updates: given $i,j,c$ increment $t_c$ between indices $i$ and $j$. After every update output $\sum_k \max{t_0[k],t_1[k]}$.

SWERC 2022 took place last week-end in Milan, the “capital of fashion”. ENS Ulm 1 won the competition, 3 teams + 1 sponsor team (Jane Street) solved 11 problems over 12; in a nutshell, the competition was fierce.

We here focus on the last problem of the practice session, Bloggers. We do not have access to the precise problem statement anymore (only the main problems), it may not be published online, so this note is based on what we remember — and we were not able to test our proposed solution. It read along these lines:

You are an influencer on a social network, and they are bloggers who indicate whether it is sexy to wear a white or black jacket during a particular time period. You can decide the color to wear on each day and you want to maximize your sexiness, i.e. the number of bloggers who think you are sexy on each day. There are $n$ days and $m$ bloggers, for each $k = 1, \ldots, m$, you are given the interval of consecutive days $[i_k, j_k]$ together with a color $c_k \in {0, 1}$ and you should output the maximal sexiness value, summed on all $n$ days, if you consider all bloggers from 1 to $k$. $n$ and $m$ may go up to $10^5$.

Formal problem

The goal is to maintain two integer tables $t_0,t_1$ of size $n$. We receive a sequence of updates of the form:

given integers $0\leq i\leq j<n$, and a bit $c$, increment $t_c$ between the indices $i$ and $j$ included.

After each update we should output the score $\sum_{k=0}^{n-1} \max(t_0[k],t_1[k])$.

In the illustration below we highlight the maximum elements among the tables for each index.

First idea

Both the table size $n$ and the number of updates $m$ are in the range of $10^5$, so it will not be efficient enough to implement the updates with a naive quadratic algorithm. An efficient data structure is needed, and the problem has very much the flavor of a segment tree at first sight.

A lazy segment tree would allow you to store an integer table, add a value to the table within a range of indices, and to query the maximum (say) inside a given range of indices. The reason why lazy segment tree works is because, once you add a value $+1$ to a range of indices $[i,j]$, then the maximum in the same range also increases by the same amount $+1$. In general you can replace the maximum by any other associative operation with this property.

If we want to adapt such a data structure to our problem, then we would need to store pairs of integers in the table, representing $(t_0[k],t_1[k])$. An update request would add $(1,0)$ or $(0,1)$ to an index range. But if we store only the sum of the maxima of the table entries in specific index ranges, then we lose important information. For example, it is important to know whether the maximum in each table cell comes from $t_0$ or from $t_1$. Indeed, this information is necessary to find out if an update increases the maximum in a cell or not.

Second idea

The second idea was given after the practice session by Marc Dufay, a competitor of an ETH Zürich team (that arrived second!). We are always deeply impressed by the short time in which the top students find solutions!

An overall complexity of $O(n + m\sqrt n)$ would be acceptable. So we could design a solution using the square root decomposition. And this is very interesting, because it is often said that the square root decomposition is a poor man’s variant to the segment tree, and it is easier to understand and implement. But it seems that the square root decomposition is sometimes more powerful.

How does it work? Let’s illustrate it on a simpler problem. We want to maintain an integer table $t$ of size $n$, such that you can increment $t$ withing a given index range, and you can query the maximum within a given index range. Both operations have complexity $O(\sqrt n)$, which is an improvement over the naive implementation with complexity $O(n)$.

The first step is to divide the table into blocks of size $B = \lfloor \sqrt n \rfloor$, except maybe the last one, which can be smaller. Second, we represent the table $t$ by a table $b$ of size $\lceil n / B \rceil$ and a table $s$ of size $n$. Each element in $b$ corresponds to a block. The interpretation is that $t[k]=b[k/B] + s[k]$ for all indices $k$. We notice that the implementation only uses the tables $b$ and $s$: $t$ is only represented implicitly.

Now you can increment $t$ within a block in constant time, simply by incrementing only the entry in $b$ which corresponds to the block. Thus, when incrementing $t$ within an index range $[i,j]$, there are three kinds of blocks:

Blocks whose range is disjoint from $[i,j]$, which are simply ignored.
Blocks whose range is included in $[i,j]$: their corresponding entry in $b$ is incremented.
Finally, blocks whose range strictly intersects $[i,j]$. The entries in $s$ belonging both to these blocks and to $[i,j]$ have to be incremented.

One can verify that an update generates only $O(\sqrt n)$ increments in total among the tables $b,s$, because there are $O(\sqrt n)$ blocks and because there are at most two blocks of the third kind.

In order to be able to answer the $\max$ requests, we can follow a similar idea. We need to maintain in addition to table $b$ a table $\textrm{score}$ of the same size, storing for each block its maximum value. Whenever $b[\ell]$ is incremented, $\textrm{score}[\ell]$ is incremented as well. And whenever $s[k]$ is incremented, the value $\textrm{score}[k/B]$ might need to be incremented as well, so to satisfy $\textrm{score}[k/B] \geq b[k/B] + s[k]$.

Designing the data structure

We follow the sqrt decomposition idea described above, and define two tables $b[c],s[c]$ for each bit $c\in\{0,1\}$. Also we have a variable $\textrm{score}$ that stores the overall score. When incrementing $s[c][k]$, we compare $t[c][k]$ and $t[1-c][k]$. If the former is strictly larger, then we know that the maximum of $t_0[k]$ and $t_1[k]$ increased as well, so in that case we also need to increase the score.

Incrementing $b[c][\ell]$ for some block $\ell$ is trickier. The score increases by the number of entries $k$ in the block for which we have $t_0[k] \geq t_1[k]$. So we need to store this number as well, in a variable $\textrm{nb_arg_max}[\ell]$. Now, this number will also change over time. Suppose that there are $x$ entries $k$ in the block with $t_0[k] = t_1[k] - 1$. After increasing $b[0][\ell]$, the variable $\textrm{nb_arg_max}[\ell]$ increases by $x$. Similarly if there are $x$ entries $k$ in the block with $t_0[k] = t_1[k]$. After increasing $b[1][\ell]$, the variable $\textrm{nb_arg_max}[\ell]$ decreases by $x$.

This means that we need to store the number of entries $k$ in the block that have a particular given difference $d=s[0][k]-s[1][k]$. Let $\textrm{nb_diff}[\ell][d]$ be this number. Here $d$ can be negative. If we use a table, we would need to shift the indices to stay with non-negative indices. For this implementation we choose to use a dictionary instead, to avoid the shift.

In summary we have the following variables. Index ranges are $0\leq c\leq 1, : 0\leq k < n,: 0\leq \ell < \lceil n / B \rceil$.

$s[c][k]$ is the first part of $t_c[k]$.
$b[c][\ell]$ is the second (block constant) part of $t_c[k]$ for all indices $k$ in the block, i.e. $\ell=k/B$.
$\textrm{nb_diff}[\ell][d]$ is the number of entries $k$ in block $\ell$ with $d=s[0][k]-s[1][k]$.
$\textrm{nb_arg_max}[\ell]$ is the number of indices $k$ in block $\ell$ with $t_0[k] \geq t_1[k]$.
- Invariant: there are $\textrm{nb_arg_max}[\ell]$ many indices $k$ in block $\ell$ with $s[0][k] + b[0][\ell] \geq s[1][k] + b[1][\ell]$.
- Reformulated invariant: $\textrm{nb_arg_max}[\ell] = \sum_d \textrm{nb_diff}[\ell][d]$ over all $d\geq b[1][\ell] - b[0][\ell]$.
$\textrm{score}$ is the overall score to be printed after each update.

The initial values are as follows.

$\textrm{score}$, $s$ and $b$ are initialized with zeroes.
$\textrm{nb_diff}[\ell][0]=B$ and $\textrm{nb_diff}[\ell][d]=0$ for all $d\neq 0$. Here we would like to use a dictionary with default value $0$ (a Counter) to represent this variable.
$\textrm{nb_arg_max}[\ell]=B$ for all blocks $\ell$.
One exception however: for the last block both variables $\textrm{nb_diff}[\ell][0]$ and $\textrm{nb_arg_max}[\ell]$ have to be initialized to the size of the last block.

For every update request with parameters $i,j,c$ we need to do the following actions. The actions are done for every block $\ell$ in the range $i/B$ to $j/B$ (included). This restriction avoids testing if the block range is disjoint from the update range $[i,j]$.

Each block $\ell$, corresponds to an index range $[u,v]$ with $u=\ell B$ and $v=\min\{n, (\ell+1)B\} - 1$.
If $i\leq u$ and $v \leq j$, then this is an included block.
- If $c=0$, we add $\textrm{nb_arg_max}$ to $\textrm{score}$. Then we increment $b[0][\ell]$ and add $\textrm{nb_diff}[\ell][d]$ to $\textrm{nb_arg_max}$ with $d=b[1][\ell] - b[0][\ell]$.
- If $c=1$, we add $v - u + 1-\textrm{nb_arg_max}+\textrm{nb_diff}[\ell][d]$ to $\textrm{score}$ with $d=b[1][\ell] - b[0][\ell]$. Next we decrement $\textrm{nb_arg_max}$ by $\textrm{nb_diff}[\ell][d]$ and then only increase $b[1][\ell]$.
- Comment: we could have made these previous steps symmetric, by storing redundantly the number of entries $k$ for which $t_0[k]\geq t_1[k]$ and the also the number of $k$ for which $t_1[k]\geq t_0[k]$. Not sure that this would not complexify some other part of the program.
Otherwise this is a partially intersecting block. In this case we loop over $k$ in the intersection of the block and the query interval $[i,j]$, that is in between $\max\{u, i\}$ to $\min\{v, j\}$.
- We decrement $\textrm{nb_diff}[\ell][d]$ for $d=s[0][k]-s[1][k]$.
- Then we increment $s[c][k]$.
- In case $s[c][k] > s[1-c][k]$, we increment $\textrm{score}$.
- And finally we increment $\textrm{nb_diff}[\ell][d]$ for $d=s[0][k]-s[1][k]$, to preserve the invariant.

Here is an illustration of one step of the algorithm. The arcs among the dictionnary entries indicate units that moved. And the arcs from the dictionnary to the table indicates an addition of the source of the arc to the target of the arc.

Implementation in Python

from collections import Counter
from math import ceil, floor, sqrt
from sys import stdin


n, m = map(int, stdin.readline().split())

# declare, initalize all variables
s = [[0 for _ in range(n)] for _ in range(2)]
B = floor(sqrt(n))
b = [[0 for _ in range(ceil(n / B))] for _ in range(2)]
# beware: in C++ n/B would be the integer division, and you need to write 1 + (n-1)/B instead

nb_diff = [Counter() for _ in range(ceil(n / B))]
nb_arg_max = [0 for _ in range(ceil(n / B))]

# we write ell instead of l because it can be mistaken with 1
for ell in range(ceil(n / B)):
    u = ell * B
    v = min((ell + 1) * B, n) - 1
    nb_diff[ell][0] = v - u + 1
    nb_arg_max[ell] = v - u + 1
score = 0


def update(c, i, j):
    global score
    for ell in range(i//B, j//B + 1):
        u =  ell * B
        v = min((ell + 1) * B, n) - 1
        d = b[1][ell] - b[0][ell]
        if i <= u and v <= j:
            if c == 0:
                score += nb_arg_max[ell]
                nb_arg_max[ell] += nb_diff[ell][d - 1]
                b[0][ell] += 1
            else:
                nb_arg_max[ell] -= nb_diff[ell][d]
                score += (v - u + 1) - nb_arg_max[ell]
                b[1][ell] += 1
        else:   # we now have u < i <= v or u <= j < v:
            for k in range(max(i, u), min(j, v) + 1):
                if s[c][k] + b[c][ell] >= s[1 - c][k] + b[1 - c][ell]:
                    score += 1
                if c == 0 and s[0][k] + b[0][ell] == s[1][k] + b[1][ell] - 1:
                    nb_arg_max[ell] += 1
                elif c == 1 and s[0][k] + b[0][ell] == s[1][k] + b[1][ell]:
                    nb_arg_max[ell] -= 1
                nb_diff[ell][s[0][k] - s[1][k]] -= 1
                s[c][k] += 1
                nb_diff[ell][s[0][k] - s[1][k]] += 1


for line in stdin:
    c, i, j = map(int, line.split())
    update(c, i - 1, j - 1)   # internally we start the indices at 0
    print(score)

Deep Reinforcement Learning

2023-01-20T00:00:00+00:00

Stable Baselines rely on TF 1.x but Stable Baselines v3 rely on PyTorch.

Breakout v4 and v5 are in the Atari Learning Environment (ale-py)
There exists a MinAtar environment that is 10x faster to train than the original Breakout (mini Atari). I fixed it in a fork so that Stable Baselines v3 can be used.
DQN of stable-baselines
Actor-Critic A2C of stable-baselines
Actor-Critic is very sensitive to hyper-parameter

Benchmarks

Best models on Breakout-v5 using CleanRL thanks to this Reddit post
Extra features of DQN, notably Rainbow
Yet another benchmark

Best model from CleanRL:

ppo_atari_envpool.py --exp-name a2c --update-epochs 1 --num-minibatches 1 --norm-adv False --num-envs 64 --clip-vloss False --vf-coef 0.25 --anneal-lr False --num-steps 5 --track.

Fun fact

Running on my CPU was faster than Colab GPU (for the MinAtar environment), possibly because the data was not high dimensional and the network was not very deep

Organizing contests with judges

2022-12-17T00:00:00+00:00

SPOJ

You need to send them a mail to get access to contest creation.

Allow languages
Access verification on view
Copy pb statement (HTML editor) / select languages / add testcases

Good points:

Very functional
Interactive evaluation is available (never used it)
Possible to have your own solution checker, but it is not that well documented (broken link; had to find it elsewhere on the forum)

Kattis

Don’t forget to start the timer otherwise people won’t be able to submit.

Good points:

Great and easy-to-use interface
Easy to set up existing problems in minutes
Also possible to have your own solution checker (don’t know how it works yet)
Possible to export maths as MathML (that can be embedded in SPOJ)

Haven’t tried to add new problems yet. Looking forward to it.

AlphaGo and Alpha Zero

2022-12-17T00:00:00+00:00

In December 2016 we presented it at ENS Paris-Saclay with Étienne Simon but apparently I never wrote a blog post in the end.

Original AlphaGo paper in Nature
Original AlphaZero paper
Very nice blog post and very nice code with a ready-to-try player on Othello 8x8 (or 6x6 trained in 3 days); however it is too weak for my current needs :/

Six dates importantes en algorithmique

2022-11-30T00:00:00+00:00

-300: Algorithme d’Euclide (qui date peut-être de l’école de Pythagore 530 av. J.C.)
820: Al-Khawarizmi “Abrégé du calcul par la restauration et la comparaison”
1670: Juan Caramuel y Lobkowitz, prélat espagnol, invente le système binaire, plagié ensuite par Leibniz en 1679
1843: la Note G d’Ada Lovelace, la première programmeuse de l’histoire (décédée à 36 ans), pour le calcul des nombres de Bernoulli sur la machine de Charles Babbage

Voir aussi cette merveilleuse lettre d’Ada à Charles :

Ada Lovelace, the first programmer, begging her collaborator Charles Babbage, the inventor of the first general computer, not to mess with her math pic.twitter.com/eofy92oL5n
— Nina Beguš (@ninabegus) September 26, 2022

1936: Turing (et 1931 Gödel, cf. Logicomix), On Computable Numbers
1953: Bellman invente la programmation dynamique. C’était un an après le premier langage de programmation Autocode (1952), ce n’était pas un programmeur, bien que Ada si, mais il avait un algo pour résoudre des problèmes de contrôle optimal, qui ont mené à l’apprentissage par renforcement. Ce qu’il a fait est vraiment fondamental en algorithmique, même si pas autant que Turing.

J’avais fait un thread de vieux articles de recherche :

Oldest research articles. A short thread. ⏬
— Jill-Jênn Vie (@jjvie) March 26, 2021

Quadrangle Inequality trick for dynamic programs

2022-11-03T00:00:00+00:00

Application: Given an ordered list of keys with frequencies, build a binary search tree on those keys which minimizes the average query cost.

This note has been updated in January 2024. We corrected one error in our implementation. See the paragraph right before the implementation for more detail.

Formal definition

In the 1970’s a technique was discovered by Knuth and generalized in the 1980’s by Yao, which permits to speedup any dynamic program of a particular form and particular property.

We illustrate the technique on the following problem. Informally, we want to construct a binary search tree, over given keys, which minimizes the average query cost.

Formally, we are given some keys numbered from $1$ to $n$, together with frequency vectors

\[\begin{array}{cccc} &\beta_1& &\beta_2& &\ldots& &\beta_n \\ \alpha_0& &\alpha_1& &\alpha_2 & &\alpha_{n-1}& &\alpha_{n} \end{array}\]

such that $\beta_i$ is the frequency of queries of the $i$-th key, $\alpha_i$ is the frequency of a query between the $i$-th and the $i+1$-th key. Here we consider fictitious keys indexed $0$ and $n+1$.

A binary search tree is a rooted tree where

every inner node is associated to a key index $i$ and has weight $\beta_i$,
inner nodes have a left and a right subtree,
every leaf has weight $\alpha_i$.

Such a tree has to satisfy the usual left-to-right ordering according to the indices of the keys.

The cost of a tree is called the weighted path length and is defined as the sum over all nodes of the weight of the node multiplied with the level of the node in the tree.

An optimal search tree can be computed using dynamic programming. For every $0\leq i\leq j\leq n$ consider the problem of building the optimal search tree for queries restricted to be strictly between the $i$-th key and the $j+1$-th key. We call it the problem restricted to $(i,j)$, or subproblem $(i,j)$. We consider the following values.

$C[i,j]$ is the cost of the optimal search tree
$W[i,j]$ is the total frequency of the restricted problem. It is $\alpha_i +\beta_{i+1}+ \alpha_{i+1} + \ldots+\beta_{j}+\alpha_j$.
$R[i,j]$ is the root of the optimal search tree. It is defined only for $i < j$ and satisfies $i+1 \leq R[i,j] \leq j$.

These values lead to the following dynamic program. For the base case $i=j$ we have

\[C[i,i] = W[i,i] = \alpha_i\]

and for $i < j$ we have

\[C[i,j] = W[i,j] + \min_{i+1\leq r\leq j} (C[i,r - 1] + C[r,j]) \\ R[i,j] = \textrm{argmin of above expression}.\]

The optimal search tree of the subproblem contains a root $i+1\leq r\leq j$, which motivates the minimum expression above. The addition of $W[i,j]$ comes from the fact that by attaching the left and right subtrees under the root $r$, the level of all their nodes increases by $1$. Since root $r$ has level one, the weight $\beta_r$ has to enter the cost of the tree as well.

This leads to a time complexity of $O(n^3)$, because we have $O(n^2)$ variables, each being the minimum over $O(n)$ alternatives.

Improvement to $O(n^2)$

Donald Knuth made this clever observation 50 years ago, that the root does not have to be searched within the full range. The following range is enough:

\[R[i,j - 1] \leq R[i,j] \leq R[i + 1,j], \tag{1}\]

and therefore

\[C[i,j] = W[i,j] + \min_{R[i,j-1]\leq r\leq R[i+1,j]} (C[i,r - 1] + C[r,j]).\]

This would mean that the time needed to compute $R[i,j]$ is proportional to $R[i,j-1] - R[i+1,j] + 1$. Summing up over all $i,j$ with fixed difference $j-i$, we obtain a telescopic sum of value $O(n)$. Hence the total time complexity is $O(n^2)$.

A general framework

Such an improvement applies under some condition to any dynamic program of the form

\[C[i,i] = 0 \\ C[i,j] = W[i,j] + \min_{i<k\leq j} (C[i,k-1]+C[k,j]) \:\:\textrm{ for } i < j.\]

This dynamic program generalizes the previous one, but for the special case $\alpha=0$. This simplifies the presentation.

The dynamic program above, in essence depends on the matrix $W$. And it is the structure of $W$, which permits the above mentioned improvement. Two properties of $W$ are essential.

$W$ satisfies the quadrangle inequality (denoted QI for short) if for every $a\leq b\leq c\leq d$ we have

\[{\color{red}{W[a,c] + W[b,d]}} \leq {\color{green}{W[b,c] + W[a,d]}}\]

$W$ is monotone on the lattice of intervals if for every $a\leq b\leq c\leq d$ we have

\[{\color{red}{W[b,c]}} \leq {\color{green}{W[a,d]}}.\]

Note: the quadrangle inequality is also called the Monge property. And it is enough that it is satisfied for $b=a+1,d=c+1$, because then it is satisfied as well for larger $b,d$.

F. Frances Yao shows that whenever $W$ satisfies the two properties, then the inequality (1) holds, which allows to solve the dynamic program in quadratic time.

This holds for a variety problems, such as

Optimal binary search tree for given query frequencies
Given sets of strings $S_1,\ldots,S_n$, compute the multi-set of strings obtained by concatenating a string from $S_1$ with a string from $S_2$ and so on and finally concatenating with a string from $S_n$. Every concatenation between two strings generates one unit of cost. The goal is to perform the task at minimum cost.
Given $n$ points in convex positions, and an integer $m < n$, compute a convex polygon using $m$ among the $n$ points which has longest perimeter.

The proof idea

For the formal proof we refer to Yao’s paper, referenced at the bottom of this document. In a nutshell it consists of the following steps.

Lemma 1 If $W$ satisfies QI and is monotone on the lattice of intervals, then $C$ also satisfies QI.

Proof

The proof of $$ C[a,c] + C[b,d] \leq C[b,c] + C[a,d] \:\:\textrm{ for all } a\leq b\leq c\leq d $$ is by induction on the difference $d-a$. When $a=b$ or $c=d$, both sides of the inequality are identical. This establishes the base case $d-a\leq 1$. The induction step considers two cases. **Case** $a<b=c<d$: In this case the inequality to show becomes the inverse triangular inequality $$ C[a,b]+C[b,d] \leq C[a,d] \:\:\textrm{ for all } a<b<d. $$ Let $k$ be the minimizer for the expression of $C[a,d]$, i.e. $C[a,d]=C_k[a,d]$, using the notation $C_k[a,d] := W[a,b] + C[a,k-1]+C[k,b]$. If $k\leq b$ we have $$ \begin{array}{rll} C[a,b]+C[b,d] &\leq C_z[a,b] + C[b,d] &\text{(by opt. of C[a,b])} \\ &= W[a,d] + C[a,k-1]+C[k,b] + C[b,d] \\ &\leq W[a,d] + C[a,k-1] + C[k,d] &\text{(by ind. hyp., using a<k)}\\ &= C[a,d]. &\text{(by choice of k)} \end{array} $$ The case $k > b$ is similar. **Case** $a<b<c<d$: Let $k,\ell$ be such that $$ C[b,c] = C_k[b,c] \textrm{ and } C[a,d] = C_\ell[a,d]. $$ If $\ell\leq k$ we have $$ \begin{array}{rll} C[a,c] + C[b,d] &\leq C_\ell[a,c] + C_k[b,d] &\text{(by opt.)} \\ &= W[a,c] + W[b,d] +C[a,\ell-1] + C[k,c] + C[b,k-1]+C[k,d] \\ &\leq W[b,c] + W[a,d] +C[a,\ell-1] + C[k,c] + C[b,k-1]+C[k,d] &\text{(by QI of W)} \\ &\leq W[b,c] + W[a,d] +C[a,\ell-1] + C[b,k-1]+C[k,c] + C[\ell,d] &\text{(by ind. hyp.)} \\ &= C_k[b,c] + C_\ell[a,d] \\ &= C[b,c] + C[a,d]. \end{array} $$ The case $\ell > k$ is similar. And this concludes the proof.

Lemma 2 If $C$ satifies QI, then

\[K[i,j] \leq K[i,j+1] \leq K[i+1,j+1] \textrm{ \:\:for } i\leq j,\]

where $K[i,j]$ is the minimizer of the minimum expression in the definition of $C[i,j]$, and for convenience we denote $K[i,i]=i$.

Proof

It holds by definition of $C$ when $i=j$. To show the first inequality in case $i < j$, we will show for $a < b\leq c < d$ $$ \left[ C_c[a,d] \leq C_b[a,d] \right] \Rightarrow \left[ C_c[a,d+1] \leq C_b[a,d+1] \right]. \tag{2} $$ By the quadrangle inequality we have $$ C[b,d]+C[c,d+1] \leq C[c,d] + C[b,d+1]. $$ And if we add $W[a,d]+W[a,d+1]+C[a,b-1]+C[a,c-1]$ to both sides we obtain $$ C_b[a,d]+C_c[a,d+1] \leq C_c[a,d]+C_b[a,d+1] $$ which shows the implication (2). The proof for the second inequality is similar.

Implementation in Python

We said earlier that the minimizer for the recursive expression for $C[i,j]$ ranges between $i+1$ and $j$, but can be restricted to the range between $K[i,j - 1]$ and $K[i + 1, j]$. This means that the actual range has to be in the intersection of these ranges, hence the use of max in the range expression for variable $k$ in the code below.

def dyn_prog_Monge(W):
    """ Solves the following dynamic program for 0 <= i < j < n

    C[i,i] = 0
    C[i,j] = W[i,j] + min over i < k <= j of (C[i,k-1] + C[k,j]) 
    K[i,j] = minimizer of above

    :param W: matrix of dimension n times n
    :assumes: W satisfies the Monge property (a.k.a. quadrangle inequality) and monotonicity in the lattice of intervals 
    :returns: C[0,n-1] and the matrix K with the minimizers
    :complexity: O(n^2)
    """
    n = len(W) 
    C = [[W[i][i] for j in range(n)] for i in range(n)] # initially C[i,i]=W[i][i]
    K = [[j for j in range(n)] for i in range(n)] # initially K[i,i]=i
    
    # recursion
    for j_i in range(1, n): # difference between j and i
        for i in range(n - j_i):
            j = i + j_i
            argmin = None
            valmin = float('+inf')
            for k in range(max(i + 1, K[i][j - 1]),  K[i + 1][j] + 1):
                alt = C[i][k - 1] + C[k][j]
                if alt < valmin:
                    valmin = alt
                    argmin = k
            C[i][j] = W[i][j] + valmin
            K[i][j] = argmin 
    return C[0][n-1], K

References

The first reference is the original paper introducing the quadratic time algorithm. There were many followup researchs, which are summarized in the second reference.

Knuth, D. E., Optimum binary search trees, Acta Informatica, 1(1), pages 14–25, 1971.
Yao, F. Frances. Efficient dynamic programming using quadrangle inequalities. Proceedings of the twelfth annual ACM symposium on Theory of computing. 1980.
Nagaraj, S. V. Optimal binary search trees, Theoretical Computer Science, 188(1–2), pages 1-44, 1997.
Our implementation of the actual code to build the optimal binary search tree

Pareto optimality

2022-11-01T00:00:00+00:00

Compute the pareto set of a given set of points in 2 or 3 dimensions.

Definition

A point $(x,y)$ dominates all points $(x’,y’)$ with $x\leq x’, y\leq y’$ where at least one inequality is strict. Given a point set the tasks is to determine all non-dominated points. These points form the so-called Pareto-set.

In 2 dimensions

In 2 dimensions we can process the points from left to right, and for points with the same x-coordinate from bottom up. This order ensures that when we process a point $p$, it cannot be dominated by points processed later.

Two important observations. If $p$ is dominated by $p’$ and $p’$ by $q$, then $p$ is also dominated by $q$. We say that domination is transitive. The result is that in order to decide if a currently processed point $p$ is dominated, we only need to check domination with the Pareto set of the already processed points. And the second observation is that if $p$ is dominated by some point $q$ in the Pareto set then it is also dominated by the last point added to the current Pareto set.

As a result we only need to compare each point with the last added point from the Pareto set.

def pareto2d(points):
    """ Compute the Pareto set of a given set of points in 2 dimensions

    :param points: list of tuples with the coordinates of the points. Can be floating point coordinates.
    :modifies: points will be sorted
    :returns: a list of non-dominated points
    :complexity: $O(n\\log n)$
    """
    pareto = []
    points.sort()
    for p in points:
        x, y = p
        if pareto == [] or y < pareto[-1][1] or p == pareto[-1]: 
            pareto.append(p)
    return pareto

The overall complexity is $O(n\log n)$ which comes from the initial sorting.

In 3 dimensions

Again we process the points in lexicographical increasing order. Every point is given by its coordinates $(x,y,z)$. For technical reasons we want to associate to $y$ its rank rank[y] among all the $y$-coordinates of the given points. This does not affect the domination order, and will allow us to use the rank as an index in a table $R$. It will be clear in a second, why this is useful.

To simplify the implementation, if there are multiple $y$-values among the given points, then we allow the ranks to be not successive. We only need them to be in order and bounded by $n$, the number of given points. For example if the $y$-values are $[1,4,4,4,6,7]$, then we will give the ranks rank[1]=0, rank[4]=3, rank[6]=4, rank[7]=5.

Again some observations. When processing point $p=(p_x,p_y,p_z)$, we only need to verify if it is dominated among the already processed points. And those already processed points have $x$-coordinate smaller equal $p_x$. So the question is if among those points there is one which has its $y$-coordinate at most $p_y$ and its $z$-coordinate at most $p_z$.

Here the table $R$ comes at hand. We store in $R[i]$ the smallest value $z$ such that we have seen a point $(x,y,z)$ with rank[y]==i. If there is no such point we set rank[y]=$+\infty$. Hence $p$ is dominated if and only if

\[\min\{R[0], R[1],\ldots, R[i]\} \leq p_z\]

for $i$ being the rank of $p_y$. A Fenwick or Segment tree can be used to store the table $R$ and allow to perform the above minimum query and update of R[i] in logarithmic time.

Small detail: if a point appears several times in the input, then its copies will be processed one after another. If it is a non-dominated point, then the second copy would appear as dominated by the above described test. But this can be handled correctly, by comparing each processed point, with the last point added to the Pareto set.

This leads to the following algorithm with an overall complexity of $O(n\log n)$.

def pareto3d(points):
    """ Compute the Pareto set of a given set of points in 2 dimensions

    :param points: list of tuples with the coordinates of the points. Can be floating point coordinates.
    :modifies: points will be sorted
    :returns: a list of non-dominated points
    :complexity: $O(n\\log n)$
    """
    # compute the ranks, it is ok to have multple y-values in the list
    y_values = [y for x,y,z in points]
    y_values.sort()
    rank = {}
    for i, yi in enumerate(y_values):
        rank[yi] = i
    n = len(points)
    points.sort()    # sort by rank in first competition
    pareto = []
    R = FenwickMin(n)
    for p in points:
        x, y, z = p 
        i = rank[y]
        if pareto == [] or R.prefixMin(i) > z or p == pareto[-1]:
            pareto.append(p) 
        R.update(i, z)
    return pareto

Fenwick Min Tree

The above code uses a variant of the Fenwick tree, which allows to store a table $R$, initially filled with $+\infty$, the neutral element for the min operator. Now we can either update $R$ at a given index $i$ and decrease its value to a given value $val$. If $R[i]$ is already smaller than $val$, then the update does nothing. And given an index $i$ we can query the minimum among $R[0],R[1],\ldots, R[i]$. All these operations can be performed in logarithmic time, in the size of the table.

class FenwickMin:
    """maintains a tree to allow quick updates and queries
    of a virtual table t
    """
    def __init__(self, size):
        """stores a table t and allows updates and queries
        of prefix sums in logarithmic time.

        :param size: length of the table
        """
        self.s = [float('+inf')] * (size + 1)  # create internal storage

    def prefixMin(self, a):
        """
        :param int a: index in t, negative a will return infinity
        :returns: min(t[0], ... ,t[a])
        """
        i = a + 1                  # internal index starts at 1
        retval = float('+inf')
        while i > 0:               # loops over neighbors
            retval = min(retval, self.s[i])    
            i -= (i & -i)          # left neighbor
        return retval

    def update(self, a, val):
        """
        :param int a: index in t
        :param val: a value
        :modifies: sets t[a] to the minimum of t[a] and val
        """
        i = a + 1                  # internal index starts at 1
        while i < len(self.s):     # loops over parents
            self.s[i] = min(self.s[i], val)       # update node
            i += (i & -i)          # parent