WARNING: The text below provides a guidance for a Project Euler problem.

• If you pride yourself in tackling problems entirely on your own, you are encouraged to exit this post right now.

• If you are simply looking for a hint for the problem, please visit the »Project Euler’s official forum«. You may find some hints here, but if you don’t want your problem spoiled, scroll cautiously.

• If you are looking for a readily available copy/paste answer, you will not find it here. All of the code snippets in the post have been stripped of some crucial lines (these are clearly marked). It is my aim that anyone able to replicate the answer based on such snippets will have to understand the solution conceptually. At that point, by virtue of learning something new, I believe he deserves it anyway.

• Guidance for future problems will not be published before 100 people have solved the problem and there are at least two more recent problems.



Constrained permutations

This problem, being one of the harder recent ones, tasks us with counting a certain type of permutations (we will call these valid). The constraint on permutations is that they should not contain any sub-permutation of type $(1243)$ and contain at most $m$ sub-permutations of type $(21)$; where the containment is defined in the problem. In line with how the containment is defined, we will consider the permutations to be, informally, finite sequences of integers. We are supposed to count valid permutations for $m=40$ and which are of length at most $10^{18}$. The single key realization that paves the path to a successful solution is that there are not that many valid permutations, and that they can be counted in a brute-force manner. This observation can be put into solid ground with the following claim:

Claim: Any valid permutation fixes all $k$ where $k \ge m + 3$.

Proof. Let $\sigma$ be a valid permutation and $x$ be the largest integer to the right of which one can find a smaller integer in $\sigma$. Then, we can find in $\sigma$ a subsequence:

where $a \ge 0$, $b \ge 1$, $a + b + 1 = x$ and all $l_i, r_j$ are smaller than $x$. Since $\sigma$ is a valid permutation and thus contains no sub-permutations of type $(1243)$, the integers $l_1, l_2, \dots, l_a$ must be in decreasing order. This means that $\sigma$ contains at the very least these sub-permutations of type $(21)$:

• All $(x, r_j)$. There is $b$ of these.
• Any pair $% $. There is $a\frac{a-1}{2}$ of these. But again, since $\sigma$ is valid, this gives us:

Coming back to how $x$ was defined, this means that for all $k \ge m+3$, there are no smaller integers to the right of $k$. From this, all $k \ge m + 3$ must be fixed. $\square$

How does this help us to brute-force the solution? Well, there is going to be only a certain number of minimal valid prefixes $p$ consisting of $1, 2, \dots, \mid p \mid$ (where the word minimal indicates that the prefix does not fix $\mid p \mid$, and that all integers greater than $\mid p \mid$ will be necessarily fixed in a permutation with $p$ as prefix). Each such prefix will then contribute to the total result with $n - \mid p \mid + 1$ valid permutations:

To construct all of the valid minimal prefixes, we can try all of the possible placements of the successive integers $1, 2, 3, \dots$, recording any present sub-permutations $(21)$ to avoid having more than $m$ of them, recording occurrences of $(12)$ to avoid any containment of sub-permutation $(1243)$, and pruning wherever possible.

The following piece of Python code gives the answer in about 17 seconds, when run with PyPy.