The perceptron model (Rosenblatt, 1958) idealizes a neuron as computing a weighted sum of its inputs (${\displaystyle {\mathit{x}}_j\in {\mathbb{R}}^N}$) based on learned input weights (${\displaystyle \mathit{w}\in {\mathbb{R}}^N}$) and applying a threshold (${\displaystyle \theta }$) to generate a binary response that is above or below threshold. A perceptron may be viewed as separating its high-dimensional input patterns into two output categories (${\displaystyle y\in \{0,1\}}$) (Figure 2A), with the categorization depending on the weights and threshold so that sufficiently weight-aligned input patterns fall into category 1 and the rest into category 0:

Figure 2

Download asset Open asset

If each partitioning of inputs into the $\{0,1\}$ categories is called a dichotomy, then the only dichotomies ‘realizable’ by a perceptron are those in which the inputs are linearly separable – that is, the set of inputs in category 0 can be separated from those in category 1 by some linear hyperplane (Figure 2). Cover’s counting theorem (Cover, 1965; Vapnik, 1998) provides a count of how many dichotomies a perceptron can realize if input patterns are random (more specifically, in general position). A set of patterns ${\displaystyle \{{\mathit{x}}_1,\dots ,{\mathit{x}}_P\}}$ in an $N$-dimensional space is in general position if no subset of size smaller than $N+1$ is affinely dependent. In other words, no subset of $n+1$ points lies in a $(n-1)$-dimensional plane for all $n\le N$. (Figure 2B) and establishes that for $P\le N$ patterns, every dichotomy is realizable by a perceptron – this is the perceptron capacity (Figure 2C). For $P=2N$, exactly half of the $2^P$ possible dichotomies are realizable; when $P\gg N$ for fixed $N$, the realizable dichotomies become a vanishing fraction of the total (Figure 2C).

Here, to characterize the place-cell scaffold, we model a place cell as a perceptron receiving grid-like inputs (Figure 1B). Across space, a particular ‘field arrangement’ is realizable by the place cell if there is some set of input weights and a threshold (Lee et al., 2020) for which its summed inputs are above threshold at only those locations and below it at all others (Figure 1A,B). We call an arrangement of exactly $K$ fields a ‘K-field arrangement.’.

In the following, we answer two distinct but related questions: (1) out of all potential field arrangements over the entire set of unique grid inputs, how many are realizable, and how does the realizable fraction differ for grid-like inputs compared to inputs with matched dimension but different structure? This is akin to perceptron function counting (Cover, 1965) with structured rather than general-position inputs and covers constraints within and across environments. We consider all arrangements regardless of sparsity, on one extreme, and $K$-field (highly sparse) arrangements on the other; these cases are analytically tractable. We expect the regime of sparse firing to interpolate between these two regimes. (2) Over what range of positions is any field arrangement realizable? This is analogous to computing the perceptron-separating capacity (Cover, 1965) for structured rather than general-position inputs.

Although the structured rather than random nature of the grid code adds complexity to our problem, the symmetries present in the code also allow for the computation of some more detailed quantities than typically done for random inputs, including capacity computations for dichotomies with a prescribed number of positive labels (K-field arrangements).

Grid cells have spatially periodic responses (Figure 1A,B). Cells in one grid module exhibit a common spatial period but cover all possible spatial phases. The dynamics of each module are low-dimensional (Fyhn et al., 2007; Yoon et al., 2013), with the dynamics within a module supporting and stabilizing a periodic phase code for position. Thus, we use the following simple model to describe the spatial coding of grid cells and modules: a module with spatial period ${\lambda }_m$ (in units of the spatial discretization) consists of ${\lambda }_m$ cells that tile all possible phases in the discretized space while maintaining their phase relationships with each other. Each grid cell’s response is a $\{0,1\}$-valued periodic function of a discretized 1D location variable (indexed by $j$); cell $i$ in module $m$ fires (has response 1) whenever $(j-i)mod{\lambda }_m=0$, and is off (has response 0) otherwise (Figure 1B). The encoding of location $j$ across all Mm modules is thus an $N$-dimensional vector ${\displaystyle {\mathit{x}}_j}$, where $N={\sum }_{m=1}^M{\lambda }_m$. Nonzero entries correspond to co-active grid cells at position $j$. The total number of unique grid patterns is ${\displaystyle L={\displaystyle \text{LCM}}(\{{\lambda }_1,\dots ,{\lambda }_M\})}$, which grows exponentially with $M$ for generic choices of the periods $\{{\lambda }_m\}$(Fiete et al., 2008). We refer to $L$ as the ‘full range’ of the code. We call the full ordered set of unique coding states ${\displaystyle \{{\mathit{x}}_j\}}$ the grid-like ‘codebook’ $X_{\mathrm{g}}$.

Because $X_{\mathrm{g}}$ includes all unique grid-like coding states across modules, it includes all possible relative phase shifts or ‘remappings’ between grid modules (Fiete et al., 2008; Monaco et al., 2011). Thus, this full-range codebook may be viewed as the union of all grid-cell responses across all possible space and environments. We assume implicitly that 2D grid modules do not rotate relative to each other across space or environments. Permitting grid modules to differentially rotate would lead to more input pattern diversity, more realizable place patterns, and bigger separating capacity than in our present computations.

The grid-like code belongs to a more general class that we call ‘modular-one-hot’ codes. In a modular-one-hot code, cells are divided into modules; within each module only one cell is allowed to be active (the within-module code is one-hot), but there are no other constraints on the code. With $m=1,\mathrm{\dots },M$ modules of sizes ${\lambda }_m$, the modular-one-hot codebook $X_{\mathrm{mo}}$ contains $P={\prod }_{m=1}^M{\lambda }_m$ unique patterns, with $P\ge L$ for a corresponding grid-like code. When $\{{\lambda }_1,\mathrm{\cdots },{\lambda }_M\}$ are pairwise coprime, $P=L$ and the grid-like and modular-one-hot codebooks contain identical patterns. However, even in this case, modular-one-hot codes may be viewed as a generalization of grid-like codes as there is no notion of a spatial ordering in the modular-one-hot codes, and they are defined without referring to a spatial variable.

Of our two primary questions introduced earlier, question (1) on counting the size of the place-cell repertoire (the number of realizable field arrangements) depends only on the geometry of the grid coding states, and not on their detailed spatial embedding (i.e., it depends on the mappings in Figure 3B–D, but not on the mapping between Figure 3A,B,D,E). In other words, it does not depend on the spatial ordering of the grid-like coding states and can equivalently be studied with the corresponding modular-one-hot code instead, which turns out to be easier. Question (2), on place-cell capacity (the spatial range $l\le L$ over which any place field arrangement is realizable), depends on the spatial embedding of the grid and place codes (and on the full chain of Figure 3A-E). For ${\displaystyle l\S lt;L}$, this would correspond to a particular rather than random subset of $X_{\mathrm{mo}}$, thus we cannot use the general properties of this generalized version of the grid-like code.

In what follows, we will contrast place field arrangements that can be obtained with grid-like or modular-one-hot codes with arrangements driven by alternatively coded inputs. To this end, we briefly define some key alternative codes, commonly encountered in neuroscience, machine learning, or in the classical theory of perceptrons. For these alternative codes, we match the input dimension (number of cells) to the modular-one-hot inputs (unless stated otherwise).

Random codes $X_{\mathrm{r}}$, used in the standard perceptron results, consist of real-valued random vectors. These are quite different from the grid-like code and all the other codes we will consider, in that the entries are real-valued rather than $\{0,1\}$-valued like the rest. A set of up to $N$ random input patterns in $N$ dimensions is linearly independent; thus, they have no structure up to this number.

Define the one-hot code $X_{\mathrm{oh}}$ as the set of vectors with a single nonzero element whose value is 1. It is a single-module version of the modular-one-hot code or may be viewed as a binarized version of the random patterns since $N$ patterns in $N$ dimensions are linearly independent. In the one-hot code, all neurons are equivalent, and there is no modularity or hierarchy.

Define the ‘binary’ code $X_{\mathrm{b}}$ as all possible binary activity patterns of $N$ neurons (Figure 4B, right). We distinguish $\{0,1\}$-valued codes from binary codes. In the binary code, each cell represents a specific position (register) according to the binary number system. Thus, each cell represents numbers at a different resolution, differing in powers of 2, and the code has no neuron permutation invariance since each cell is its own module; thus, it is both highly hierarchical and modular.

Figure 4

Download asset Open asset

The grid-like and modular-one-hot codes exhibit an intermediate degree of modularity (multiple cells make up a module). If the modules are of a similar size, the code has little hierarchy.

We first explore question $(1)$. The modular-one-hot codebook $X_{\mathrm{mo}}$ is invariant to permutations of neurons (input matrix rows) within modules, but rows cannot be swapped across modules as this would destroy the modular structure. It is also invariant to permutations of patterns (input matrix columns ${\displaystyle {\mathit{x}}_j}$). Further, the codebook includes all possible combinations of states across modules, so that modules function as independent encoders. These symmetries are sufficient to define the geometric arrangement of patterns in $X_{\mathrm{mo}}$, and the geometry in turn will allow us to count the number of field arrangements that are realizable by separating hyperplanes.

To make these ideas concrete, consider a simple example with module sizes $\{2,3\}$ (corresponding to the periods in the grid-like code), as in Figure 1B and Figure 3B. Independence across modules causes the code to have a product structure in the code: the codebook consists of six states that can be obtained as products of the within-module states: ${\displaystyle \{10100,10010,10001,01100,01010,01001\}}$ = ${\displaystyle \{10,01\}\times \{100,010,001\}}$, where $\{10,01\}$ and $\{100,010,001\}$ are the coding states within the size-2 and size-3 modules, respectively. We represent the two states in the size-2 module by two vertices, connected by an edge, which shows allowed state transitions within the module (Figure 4A, right). Similarly, the three states in the size-3 module and transitions between them are represented by a triangular graph (Figure 4A, right). The product of this edge graph and the triangle graph yields the full codebook $X_{\mathrm{mo}}$. The resulting product graph (Figure 4A, left) is an orthogonal triangular prism with vertices representing the combined patterns.

This geometric construction generalizes to an arbitrary number of modules $M$ and to arbitrary module sizes (periods) ${\lambda }_m$, $1\le m\le M$: by permutation invariance of neurons within modules, and independence of modules, the patterns of the codebook $X_{\mathrm{mo}}$ and thus of the corresponding grid-like codebook $X_{\mathrm{g}}$ always lie on the vertices of some convex polytope (e.g., the triangular prism), given by an orthogonal product of $M$ simplicies (e.g., the line and triangle graphs). Each simplex represents one of the modules, with simplex dimension ${\lambda }_m-1$ for module size (period) ${\lambda }_m$ (see Place-cell capacity and volatility with grid-like inputs).

This geometric construction provides some immediate results on counting: in a convex polytope, any vertex can be separated from all the rest by a hyperplane; thus, all one-field arrangements are realizable. Pairs of vertices can be separated from the rest by a hyperplane if and only if the pair is directly connected by an edge (Figure 3D). Thus, we can now count the set of all realizable two-field arrangements as the number of adjacent vertices in the polytope. Unrealizable two-field arrangements, which consist geometrically of positive labels assigned to nonadjacent vertices, correspond algebraically to firing fields that are not separated by integer multiples of either of the grid periods (Figure 3D,E).

Moreover, note that the convex polytopes obtained for the grid-like code remain qualitatively unchanged in their geometry if the nonzero activations within each module are replaced by graded tuning curves as follows: convert all neural responses within a module into graded values by convolution along the spatial dimension by a kernel that has no periodicity over distances smaller than the module period (thus, the kernel cannot, for instance, be flat or contain multiple bumps within one module period). This convolution can be written as a matrix product with a circulant matrix of full rank and dimension equal to the full range $L$. Thus, the rank of the convolved matrix ${\tilde{X}}_g$ remains equal to the rank of $X_g$. Moreover, ${\tilde{X}}_g$ maintains the modular structure of $X_g$: it has the same within-module permutation invariance and across-module independence. Thus, the resulting geometry of the code – that it consists of convex polytopes constructed from orthogonal products of simplices – remains unchanged. As a result, all counting derivations, which are based on these geometric graphs, can be carried out for $\{0,1\}$-valued codes without any loss of generalization relative to graded tuning curves. (However, the conversion to graded tuning will modify the distances between vertices and thus affect the quantitative noise robustness of different field arrangements, as we will investigate later.) Later, we will also show that the counting results generalize to higher dimensions and higher-resolution phase representations within each module.

Given this geometric characterization of the grid-like and modular-one-hot codes, we can now compute the number of realizable field arrangements it is possible to obtain with separating hyperplanes.

For modular-one-hot codes (but not for random codes), it is possible to specify any separating hyperplane using only non-negative weights and an appropriate threshold. This is an interesting property in the neurobiological context because it means that the finding that projections from entorhinal cortex to hippocampus are excitatory (Steward and Scoville, 1976; Witter et al., 2000; Shepard, 1998) does not further constrain realizable field arrangements.

It is also an interesting property mathematically, as we explore below: combined with the within-module permutation invariance property of modular-one-hot codes, the non-negative weight observation allows us to map the problem onto Young diagrams (Figure 5), which enables two things: (1) to move from considering separating hyperplanes geometrically, where infinitesimal variations represent distinct hyperplanes even if they do not change any pattern classifications, to considering them topologically, where hyperplane variations are considered as distinct only if they change the classification of any patterns, and (2) to use counting results previously established for Young diagrams.

Figure 5

Download asset Open asset

Let us consider the field arrangements permitted by combining grid-like inputs from two modules, of periods ${\lambda }_1$ and ${\lambda }_2$, (Figure 5A). The total number of distinct grid-cell modules is estimated to be between 5 and 8 (Stensola et al., 2012). Further, there is a spatial topography in the projection of grid cells to the hippocampus, such that each local patch of the hippocampus likely receives inputs from 2, and likely no more than 3, grid modules (Witter and Groenewegen, 1984; Amaral and Witter, 1989; Witter and Amaral, 1991; Honda et al., 2012; Witter et al., 2000). We denote cells by their outgoing weights ($w_{ij}$ is the weight from cell $j$ in module $i$) and arrange the weights along the axes of a coordinate space, one axis per module, in order of increasing size (Figure 5B). Since modular-one-hot codes are invariant to permutation of the cells within a module, we can assume a fixed ordering of cells and weights in counting all realizable arrangements, without loss of generality. The threshold (dark purple line) sets which combination of summed weights can contribute to a place field arrangement: no cell combinations below the boundary (purple region) have too small a summed weight and cannot contribute, while all cell combinations with larger summed weights (white region) can (Figure 5B). Decreasing the threshold (from Figure 5B to C) or increasing weights (from Figure 5B,C to D) a sufficient amount so some cells cross the threshold increases the number of combinations. But changes that do not cause cells to move past the threshold do not change the combinations (Figure 5B, solid versus dashed gray lines).

Young diagrams extract this topological information, stripping away geometric information about analog weights (Figure 5E). A Young diagram consists of stacks of blocks in rows of nonincreasing width, with maximum width and height given in this case by the two module periods, respectively. The number of realizable field arrangements turns out to be equivalent to the total number of Young diagrams that can be built of the given maximum height and width (see Appendix 3). With this mapping, we can leverage combinatorial results on Young diagrams (Fulton and Fulton, 1997; Postnikov, 2006) (commonly used to count the number of ways an integer can be written as a sum of non-negative integers).

As a result, the total number of separating hyperplanes (K-field arrangements for all $K$) across the full range $L$ can be written exactly as (see Appendix 3).

where $S_k^{(n)}$ are Stirling numbers of the second kind and $B_k^{(n)}$ are the poly-Bernoulli numbers (Postnikov, 2006; Kaneko, 1997). Assuming that the two periods have a similar size $({\lambda }_1\approx {\lambda }_2\equiv \lambda )$, this number scales asymptotically as (de Andrade et al., 2015).

Thus, the number of realizable field arrangements with $\sim {\lambda }^2$ distinct modular-one-hot input patterns in a $2\lambda$-dimensional space grows nearly as fast as ${\lambda }^{2\lambda }$, (Table 1, row 2, columns 1–3). The total number of dichotomies over these input patterns scales as $2^{{\lambda }^2}.$ Thus, while the number of realizable arrangements over the full range is very large, it is a vanishing fraction of all potential arrangements (Table 1, row 2, column 4).

If $M\ge 3$ modules were to contribute to each place field’s response, then all realizable field arrangements still would correspond to Young diagrams; however, not all diagrams would correspond to realizable arrangements. Thus, counting Young diagrams would yield an upper bound on the number of realizable field arrangements but not an exact count (see Appendix 3). The latter limitation is not a surprise: Due to the structure of the grid-like code (a product of simplices), the enumeration of realizable dichotomies with arbitrarily many input modules is expected to be at least as challenging as that of Boolean functions. Counting the number of linearly separable Boolean functions of arbitrary (input) dimension (Peled and Simeone, 1985; Hegedüs and Megiddo, 1996) is hard.

Nevertheless, we can provide an exact count of the number of realizable $K$-dichotomies for arbitrarily many input modules $M$ if $K$ is small ($K=1,2,3$ and 4). This may be biologically relevant since place fields tend to fire sparsely even on long tracks and across environments. In this case, the number ${\displaystyle {\mathcal{N}}_K}$ of realizable small-$K$ field arrangements scales as (the exact expression is derived analytically in Appendix 3)

The scaling approximation becomes more accurate for periods that are large relative to the spatial discretization (see Appendix 3). Since the total number of K-dichotomies scales as ${\lambda }^{MK}$, the fraction of realizable K-dichotomies scales as ${(M/\lambda )}^{K-1}{\lambda }^{-(M-1)}$, which for ${\displaystyle \lambda \gg 1,\lambda \S gt;M}$ vanishes as a power law as soon as ${\displaystyle M\S gt;1}$.

We can compare this result with the number of K-field arrangements realizable by one-hot codes. Since any arrangement is realizable with one-hot codes, it suffices to simply count all K-field arrangements. The full range of a one-hot code with $M\lambda$ cells is $M\lambda$, thus the number of realizable K-field arrangements is ${\displaystyle {\mathcal{N}}_K=(\genfrac{}{}{0ex}{}{M\lambda }{K})\sim (M\lambda {)}^K}$, where the last scaling holds for $K\ll M\lambda$. In short, a one-hot code enables $\sim M^K{\lambda }^K$ arrangements, while the corresponding modular-one-hot code with $M\lambda$ cells enables $\sim M^{K-1}{\lambda }^{K+M-1}$ field arrangements, for a ratio ${\lambda }^{M-1}/M\gg 1$ of realizable fields with modular-one-hot versus one-hot codes. Once again, as in the case where we counted arrangements without regard to sparseness, the grid-like code enables far more realizable K-field arrangements than one-hot codes.

In summary, place cells driven by grid inputs can achieve a very large number of unique coding states that grows exponentially with the number of modules. We have derived this result for $M=2$ and all K-field arrangements, on one hand, and for arbitrary $M$ but ultra-sparse (small-$K$) field arrangements. It is difficult to obtain an exact result for sparse field arrangements for which $K$ is a small but finite fraction of $L$; however, we expect that regime should interpolate between these other two; it will be interesting and important for future work to shed light on this intermediate regime. In all cases, the number of realizable arrangements is large but a vanishingly small fraction of all arrangements, and thus forms a highly structured subset. This suggests that place cells, when driven by grid-cell inputs, can form a very large number of field arrangements that seem essentially unrestricted, but individual cells actually have little freedom in where to place their fields.

How does the number of realizable place field arrangements differ for input codes with different levels of modularity and hierarchy? We directly compare codes with the same neuron budget (input dimension $N$) by taking $N=M\lambda$, where for simplicity, we set ${\lambda }_i=\lambda$ for all modules in the modular-one-hot codes. This is because the modular-one-hot codes include all permutations of states in each module, the number of unique input states with equal-sized modules still equals the product of periods $L={(N/M)}^M={\lambda }^M$, as when the periods are different and coprime. The one-hot code generates far fewer distinct input patterns ($L=N=M\lambda$) than the modular-one-hot code, which in turn generates fewer input patterns than the binary code ($L=2^N=2^{M\lambda }$) (Table 1, column 2). This is due to the greater expressive power afforded by modularity and hierarchy.

Next, we compare results across codes for $M=2$, the case for which we have an explicit formula counting the total number of realizable field arrangements for any $K$, and which is also best supported by the biology.

How many dichotomies are realizable with these inputs? As for the modular-one-hot codes, the patterns of $X_{\mathrm{oh}}$ and $X_{\mathrm{b}}$ fall on the vertices of a convex polytope. For $X_{\mathrm{oh}}$, that polytope is just a $(N-1)$-dimensional simplex (Figure 4C, left), thus any subset of $K$ vertices ($1\le K\le N$) lies on a $(K-1)$-dimensional face of the simplex and is therefore a linearly separable dichotomy. Thus, all $2^N$ dichotomies of $X_{\mathrm{oh}}$ are realizable and the fraction of realizable dichotomies is 1 (Table 1, columns 3 and 4). For $X_{\mathrm{b}}$, the polytope is a hypercube; it therefore consists of square faces, a prototypical configuration of points not in general position (not linearly separable, Figure 2B and Figure 4, right) even when the number of patterns is small relative to the input dimension (number of cells). Counting the number of linearly separable dichotomies on vertices of a hypercube (also called linear Boolean functions) has attracted much interest (Peled and Simeone, 1985; Hegedüs and Megiddo, 1996). It is an NP-hard combinatorial problem, so no exact solution exists. However, in the limit of large dimension ($N\to \mathrm{\infty }$), the number of linearly separable dichotomies scales as $2^{N^2/2}$(Zuev, 1989), a much larger number than for one-hot inputs (Table 1, column 3). However, this number is a strongly vanishing fraction of all $2^{2^N}$ hypercube dichotomies (Table 1, column 4).

For modular-one-hot codes with $M$ modules, the polytopes contain $M$-dimensional hypercubes and not all patterns are thus in general position. We determined earlier that the total number of realizable dichotomies with $M=2$ modules scales as ${\lambda }^{2\lambda }$, permitting a direct comparison with the one-hot and binary codes (Table 1, row 2).

Finally, we may compare grid-like codes with random (real-valued) codes, which are the standard inputs for the classical perceptron results. For a fixed input dimension, it is possible to generate infinitely many real-valued patterns, unlike the finite number achievable by $\{0,1\}$-valued codes. We thus construct a random codebook $X_{\mathrm{r}}$ with the same number, $P={\lambda }^2$, of input patterns as the modular-one-hot code. We then determine the input dimension $N$ required to obtain the same number of realizable field arrangements as the grid-like code. The number of realizable dichotomies of the random code with $P\gg N$ patterns scales as $P^N\sim {\lambda }^{2N}$ according to an asymptotic expansion of Cover’s function counting theorem (Cover, 1965). For this number to match $\sim {\lambda }^{2\lambda }$, the number of realizable field arrangements with a one-hot-modular code (of two modules of size $\sim \lambda$ each requires) $N\sim \lambda$. This is a comparable number of input cells in both codes, which is an interesting result because unlike for random codes the grid-like input patterns are not in general position, the states are confined to be $\{0,1\}$-valued, and the grid input weights can be confined to be non-negative.

In sum, the more modular a code, the larger the set of realizable field arrangements, but these are also increasingly special subsets of all possible arrangements and are strongly structured by the inputs, with far from random or arbitrary configurations. Modular-one-hot codes are intermediate in modularity. Therefore, grid-driven place-cell responses occupy a middle ground between pattern richness and constrained structure.

We now turn to question (2) from above: what is the maximal range of locations, $l^{*}$, over which all field arrangements are realizable? Once we reference a spatial range, the mapping of coding states to spatial locations matters (specifically, the fact that locations in the range are spatially contiguous matters, but given the fact that the code is translationally invariant [Fiete et al., 2008], the origin of this range does not). We thus call $l^{*}$ the ‘contiguous-separating capacity’ of a place cell (though we will refer to it as separating capacity, for short); it is the analogue of Cover’s separating capacity (Cover, 1965), but for grid-like inputs with the addition of a spatial contiguity constraint.

We provide three primary results on this question. (1) We establish that for grid-structured inputs, the separating capacity $l^{*}$ equals the rank $R$ of the input matrix. (2) We establish analytically a formula for the rank $R$ of grid-like input matrices with integer periods and generalize the result to real-valued periods. (3) We show that this rank, and thus the separating capacity for generic real-valued module periods, asymptotically approaches the sum $\mathrm{\Sigma }\equiv {\sum }_{m=1}^M{\lambda }_m$. Our results are verified by numerical simulation and counting (proofs provided in Supporting Information Appendix).

We begin with a numerical example, using periods {3,4} (Figure 6A): the full range is $L=12$, while we see numerically that the contiguous-separating capacity is $l^{*}=6$. Although the separating capacity with grid-structured inputs is smaller than with random inputs, it is notably not much smaller (Figure 6B, black versus cyan curves), and it is actually larger than for random inputs if the read-out weights are constrained to be non-negative (Figure 6B, pink curves). Later, we will further show that the larger random-input capacity of place cells with unrestricted weights comes at the price of less robustness: the realizable fields have smaller margins. Next, we analytically characterize the separating capacity of place cells with grid-like inputs.

Figure 6

Download asset Open asset

For inputs in general position, the separating capacity equals the rank of the input matrix (plus 1 when the threshold is allowed to be nonzero), and the rank equals the dimension (number of cells) of the input patterns – the input matrix is full rank. When inputs are in general position, all input subsets of size equaling the separating capacity have the same rank. But when input patterns are not in general position, some subsets can have smaller ranks than others even when they have the same size. Thus, when input patterns are not in general position the separating capacity is only upper bounded by the rank of the full input matrix. In turn, the rank is only upper bounded by the number of cells (the input matrix need not be full rank).

For the grid-like code, all codewords can be generated by the iterated application of a linear operator $J$ to a single codeword: a simultaneous one-unit phase shift by a cyclic permutation in each grid module is such an operator $J$, which can be represented by a block-form permutation matrix. The sequence ${\displaystyle \mathit{x},J\mathit{x},J^2\mathit{x},\dots J^m\mathit{x}}$ of patterns generated by applying $J$ to a grid-like codeword ${\displaystyle \mathit{x}}$ with the same module structure represents $m$ contiguous locations (Figure 6C).

The separating capacity for inputs generated by iterated application of the same linear operation saturates its bound by equaling the rank of the input pattern matrix. Since a code ${\displaystyle \mathit{x},J\mathit{x},J^2\mathit{x},J^3\mathit{x},\dots }$, generated by some linear operator $J$ with starting codeword ${\displaystyle \mathit{x}}$ is translation invariant, the number of dimensions spanned by these patterns strictly increases until some value $l$, after which the dimension remains constant. By definition, $l$ is therefore the rank $R$ of the input pattern matrix. It follows that any contiguous set of $l=R$ patterns is linearly independent, and thus in general position, which means that the separating capacity of such a pattern matrix is $R$.

For place cells, it follows that whenever $l\le R$, with $R$ the rank of the grid-like input matrix, all field arrangements are realizable, while for any ${\displaystyle l\S gt;R}$, there will be nonrealizable field arrangements (Supporting Information Appendix). Therefore, the contiguous-separating capacity for place cells is $l^{*}=R$. This is an interesting finding: the separating capacity of a place cell fed with structured grid-like inputs approaches the same capacity as if fed with general-position inputs of the same rank. Next, we compute the rank $R$ for grid-like inputs under increasingly general assumptions.

We have already argued that our counting arguments hold for realistic tuning curve shapes with graded activity profiles. This follows from the fact that convolution of the grid-like codes with appropriate smoothing kernels does not change the general geometric arrangement of codewords relative to each other as these convolution operations preserve within-module permutation symmetries and across-module independence in the code. We have also shown that the contiguous-separating capacity results apply to real-valued grid periods with dense phase encodings within each module.

Here, we describe the generalization to different spatial dimensions. Consider a $d$-dimensional grid-like code consisting of ${({\lambda }_m)}^d$ cells in the mth module to produce a one-hot phase code for ${\lambda }_m$ (discrete) positions along each dimension (Figure 6E). Since the counting results rely only on the existence of a modular-one-hot code and not any mapping from real spaces to coding states, this code across multiple modules $m=1,\mathrm{\dots },M$ is equivalent to a modular-one-hot coding for ${\prod }_{m=1}^M{({\lambda }_m)}^d$ states, with modules of size ${({\lambda }_m)}^d$ each. All the counting results from before therefore hold, with the simple substitution ${\lambda }_m\to {({\lambda }_m)}^d$ in the various formulae.

The contiguous-separating capacity in $d$-dimensions is defined as the maximum volume over which all field arrangements are realizable. Like the 1D separating capacity results, this volume depends upon the mapping of physical space to grid-like codes. We are able to show that for grid modules with periods ${\lambda }_1,\mathrm{\dots },{\lambda }_M$ the generalized separating capacity is $l_d^{\star }={\mathrm{\Sigma }}_d={\sum }_{m=1}^M{\lambda }_m^d$ (see Section B.4; Figure 6F). This result follows from essentially the same reasoning as for 1D environments, but with the use of $d$-dimensional phase-shift operators.

An important quality of field arrangements that is neglected when merely counting the number of realizable arrangements or determining the separating capacity is robustness: these computations consider all realizable field arrangements, but field arrangements are practically useful only if they are robust so that small amounts of perturbation or noise in the inputs or weights do not render them unrealizable. Above, we showed that grid-like codes enable many dichotomies despite being structurally constrained, but that random analog-valued codes as well as more hierarchical codes permit even more dichotomies. Here, we show that the dichotomies realized by grid codes are substantially more robust to noise and thus more stable.

The robustness of a realizable dichotomy in a perceptron is given by its margin: for a given linear decision boundary, the margin is the smallest datapoint-boundary distance for each class, summed for the two classes. The maximum margin is the largest achievable margin for that dataset. The larger the maximum margin, the more robust the classification. We thus compare maximum margins (herein simply referred to as margins) across place field arrangements, when the inputs are grid-like or not.

Perceptron margins can be computed using quadratic programming on linear support vector machines (Platt, 1998). We numerically solve this problem for three types of input codes (permitting a nonzero threshold and imposing no weight constraints): the grid-like code $X_{\mathrm{g}}$; the shuffled grid-like code $X_{\mathrm{gs}}$ – a row- and column-shuffled version of the grid-like code that breaks its modular structure; and the random code $X_{\mathrm{r}}$ of uniformly distributed random inputs (Figure 7). To make distance comparisons meaningful across codes, $(1)$ all patterns (columns) involve the same number of neurons (dimension), $(2)$ have the same total activity level (unity L₁ norm), and $(3)$ the number of input patterns is the same across codes, and chosen to equal $L$, the full range of the corresponding grid-like code. To compute margins, we consider only the realizable dichotomies on these patterns.

Figure 7

Download asset Open asset

The margins of all realizable place field arrangements with grid-like inputs are shown in Figure 7A (black); the margin values for all arrangements are discretized because of the geometric arrangements of the inputs, and each black bar has a very high multiplicity. The grid-like code produces much larger-margin field arrangements than shuffled versions of the same code and random codes (Figure 7A, pink and blue). The higher margins of the grid-like compared to the shuffled grid-like code show that it is the structured geometry and modular nature of the code that produce well-separated patterns in the input space (Figure 4B) and create wide margins and field stability. In other words, place field arrangements formed by grid inputs, though smaller in number than arrangements with differently coded inputs, should be more robust and stable against potential noise in neural activations or weights.

Next, we directly consider how different kinds of nongrid inputs, driving place cells in conjunction with grid-like inputs, affect our results on place field robustness. We examine two distinct types of added nongrid input: (1) spatially dense noise that is meant to model sources of uncontrolled variation in inputs to the cell and (2) spatially sparse and reliable cues meant to model spatial information from external landmarks.

After the addition of dense noise, previously realizable grid-driven place field arrangements remain realizable and their margins, though somewhat lowered, remain relatively large (Figure 7B, empty green violins). In other words, grid-driven place field arrangements are robust to small, dense, and spatially unreliable inputs, as expected given their large margins. Note that because the addition of dense i.i.d. noise to grid-like input patterns pushes them toward general position, and general-position inputs enable more realizable arrangements, the noise-added versions of grid-like inputs also give rise to some newly realizable field arrangements (Figure 7B, full green violins). However, as with arrangements driven purely by random inputs, these new arrangements have small margins and are relatively not robust. Moreover, since by definition noise inputs are assumed to be spatially unreliable, the newly realizable arrangements will not persist across trials.

Next, the addition of sparse spatial inputs (similar to the one-hot codes of Table 1, though the sparse inputs here are nearly but not strictly orthogonal) leaves previous field arrangements largely unchanged and their margins substantially unmodified (Figure 7C, empty green violins). In addition, a few more field arrangements become realizable and these new arrangements also have large margins (Figure 7C, full green violins). Thus, sufficiently sparse spatial cues can drive additional stable place fields that augment the grid-driven scaffold without substantially modifying its structure. Plasticity in weights from these sparse cue inputs can drive the learning of new fields without destabilizing existing field arrangements.

In sum, grid-driven place arrangements are highly robust to noise. Combining grid-cell drive with cue-driven inputs can produce robust maps that combine internal scaffolds with external cues.

Our results on the fraction of realizable place field arrangements and on place-cell-separating capacity with grid-like inputs imply that place cells have highly restricted flexibility in laying down place fields (without direct drive from external spatially informative cues) over distances greater than $\mathrm{\Sigma }$, the sum of the input grid module periods. Selecting an arrangement of fields over this range then constrains the choices that can be made over all remaining space in the same environment and across environments. Conversely, changing the field arrangement in any space by altering the grid-place weights should affect field arrangements everywhere.

We examine this question quantitatively by constructing realizable K-field arrangements (with grid-like responses generated as 1D slices through 2D grids [Yoon et al., 2016]), then attempting to insert one or a few new fields (Figure 8A,B). Inserting even a single field at a randomly chosen location through Hebbian plasticity in the grid-place weights tends to produce new additional fields at uncontrolled locations, and also leads to the disappearance of existing fields (Figure 8A,B).

Figure 8

Download asset Open asset

Interestingly, though field insertion affects existing arrangements through the uncontrolled appearance or disappearance of other fields, it does not tend to produce local horizontal displacements of existing fields (Figure 8C): fields that persist retain their firing locations or they disappear entirely, consistent with the surprising finding of a similar effect in experiments (Ziv et al., 2013).

The locations of fields, including of uncontrolled field additions, are well-predicted by the structure (autocorrelation) of that cell’s grid inputs (Figure 8D). This multi-peaked autocorrelation function, with large separations between the tallest peaks, reflects the multi-periodic nature of the grid code and explains why fields tend to appear or disappear at remote locations rather than shifting locally: modest weight changes in the grid-like inputs modestly alter the heights of the peaks, so that some of the well-separated tall peaks fall below threshold for activation while others rise above.

Quantitatively, insertion of a single field at an arbitrary location in a 20 m span grid-place weight plasticity results in the insertion or deletion, on average, of $\sim 0.2$ uncontrolled fields per meter. The insertion of four fields anywhere over 20 m results in an average of one uncontrolled field per meter (Figure 8E).

Thus, if a place cell were to add a field in a new environment or within a large single environment by modifying the grid-place weights, our results imply that it is extremely likely that this learning will alter the original grid-cell-driven field arrangements (scaffold). By contrast, adding fields that are driven by spatially specific external cues, though plasticity in the cue input-to-place cell synapses, may not affect field arrangements elsewhere if the cues are sufficiently sparse (unique); in this case, the added field would be a ‘sensory’ field rather than an internally generated or ‘mnemonic’ one.

In sum, the small separating capacity of place cells according to our model may provide one explanation for the high volatility of the place code across tens of days (Ziv et al., 2013) if grid-place weights are subject to any plasticity over this timescale. Alternatively, to account for the stability of spatial representations over shorter timescales, our results suggest that external cue-driven inputs to place cells can be plastic but the grid-place weights, and correspondingly, the internal scaffold, may be fixed rather than plastic (Figure 8F). In experiments that induce the formation of a new place field through intracellular current injection (Bittner et al., 2015), it is notable that the precise location of the new field was not under experimental control: potentially, an induced field might only be able to form where an underlying (near-threshold) grid scaffold peak already exists to help support it, and the observed long plasticity window could enable place cells to associate a plasticity-inducing cue with a nearby scaffold peak.

This alternative is consistent with the finding that entorhinal-hippocamapal connections stabilize long-term spatial and temporal memory (Brun et al., 2008; Brun et al., 2002; Suh et al., 2011).

Finally, we note that the robustness of place field arrangements obtained with grid-like inputs is not inconsistent with the volatility of field arrangements to the addition or deletion of new fields through grid-place weight plasticity. Grid-driven place field arrangements are robust to random i.i.d. noise in the inputs and weights, as well as the addition of nongrid sparse inputs. On the other hand, the volatility results involve associative plasticity that induces highly nonrandom weight changes that are large enough to drive constructive interference in the inputs to add a new field at a specific location. This nonrandom perturbation, applied to the distributed and globally active grid inputs, results in global output changes.

We showed that when driven by grid-like inputs, place cells can generate a spatial response scaffold that is influenced by the structural constraints of the grid-like inputs. Because of the richness of their grid-like inputs, individual place cells can generate a large library of spatial responses; however, these responses are also strongly structured so that the realizable spatial responses are a vanishingly small fraction of all spatial responses over the range where the grid inputs are unique. However, realizable spatial field arrangements are robust, and place cells can then ‘hang’ external sensory cues onto the spatial scaffold by associative learning to form distinct maps spatial maps for multiple environments. Note that our results apply equally well to the situation where grid states are incremented based on motion through arbitrary Euclidean spaces, not just spatial ones (Killian et al., 2012; Constantinescu et al., 2016; Aronov et al., 2017; Klukas et al., 2020).

Mathematically, formulating the problem of place field arrangements as a perceptron problem led us to examine the realizable (linearly separable) dichotomies of patterns that lie not in general position but on the vertices of convex regular polytopes, thus extending Cover’s results to define capacity for a case with geometrically structured inputs (Cover, 1965). Input configurations not in general position complicate the counting of linearly separable dichotomies. For instance, counting the number of linearly separable Boolean functions, which is precisely the problem of counting the linearly separable dichotomies on the hypercube, is NP-hard (Peled and Simeone, 1985; Hegedüs and Megiddo, 1996).

We showed that the geometry of grid-cell inputs is a convex polytope, given by the orthogonal product of simplices whose dimensions are set by the period of each grid module divided by the resolution. Grid-like codes are a special case of modular-one-hot codes, consisting of a population divided into modules with only one active cell (group) at a time per module.

Exploiting the symmetries of modular-one-hot codes allowed us to characterize and enumerate the realizable K-field arrangements for small fixed $K$. Our analyses relied on combinatorial objects called Young diagrams (Fulton and Fulton, 1997). For the special case of $M=2$ modules, we expressed the number of realizable field arrangements exactly as a poly-Bernoulli number (Kaneko, 1997). Note that with random inputs, by contrast, it is not well-posed to count the number of realizable K-field arrangements when $K$ is fixed since the solution will depend on the specific configuration of input patterns. While we have considered two extreme cases analytically, one with no constraints on place field sparsity and the other with very few fields, it remains an outstanding question of interest to examine the case of sparse but not ultra-sparse field arrangements in which the number of fields is proportional to the full range, with a constant small prefactor (Itskov and Abbott, 2008). Finding results in this regime would involve restricting our count of all possible Young diagrams to a subset with a fixed filled-in area (purple area in Figure 5). This constraint makes the counting problem significantly harder.

We showed using analytical arguments that our results generalize to analog or graded tuning curves, real-valued periods, and dense phase representations per module. We also showed numerically that our qualitative results hold when considering deviations from the ideal, like the addition of noise in inputs and weights. The relatively large margins of the place field arrangements obtained with grid-like inputs make the code resistant to noise. In future work, it will be interesting to further explore the dependence of margins, and thus the robustness of the place field arrangements, on graded tuning curve shapes and the phase resolution per module.

As described in the section on separating capacity, once grid-place weights are set over a relatively small space (about the size of the sum of the grid module periods), they set up a scaffold also outside of that space (within and across environments). Associating an external cue with this scaffold would involve updating the weights from the external sensory inputs to place cells that are close to or above threshold based on the existing scaffold. This does not require relearning grid-place weights and does not cause interference with previously learned maps.

By contrast, relearning the grid-place weights for insertion of another grid-driven field rearranges the overall scaffold, degrading previously learned maps (volatility: Ziv et al., 2013). If we consider a realizable field arrangement in a small local region of space then impose some desired field arrangement in a different local region of space through Hebbian learning, we might ask what the effect would be in the first region. Our results on field volatility provide an answer: if the first local region is of a size comparable to the sum of the place cell’s input grid periods, then any attempt to choose field locations in a different local region of space (e.g., a different environment) will almost surely have a global effect that will likely affect the arrangement of fields in the first region. A similar result might hold true if the first region is actually a disjoint set of local regions whose individual side lengths add up to the sum of input grid periods. This prediction might be consistent with the observed volatility of place fields over time even in familiar environments (Ziv et al., 2013).

Our volatility results alternatively raise the intriguing possibility that grid-place weights, and thus the scaffold, might be largely fixed and not especially plastic, with plasticity confined to the nongrid sensory cue-driven inputs and in the return projections from place to grid cells. The experiments of Rich et al., 2014 – in which place cells are recorded on a long track, the animal is then exposed to an extended version of the track, but the original fields do not shift – might be consistent with this alternative possibility. These are two rather strong and competing predictions that emerge from our model, each consistent with different pieces of data. It will be very interesting to characterize the nature of plasticity in the grid-to-place weights in the future.

This work models place cells as feedforward-driven conjunctions between (sparse) external sensory cues and (dense) motion-based internal position estimates computed in grid cells and represented by multi-periodic spatial tuning curves. In considering place-cell responses as thresholded versions of their feedforward inputs including from grid cells, our model follows others in the literature that make similar assumptions (Hartley et al., 2000; Solstad et al., 2006; Sreenivasan and Fiete, 2011; Monaco et al., 2011; Cheng and Frank, 2011; Whittington et al., 2020). These models do not preclude the possibility that place cells feed back to correct grid-cell states, and some indeed incorporate such return projections (Sreenivasan and Fiete, 2011; Whittington et al., 2020; Agmon and Burak, 2020). It will be interesting in future work to analyze how such return projections affect the capacity of the combined system.

Our assumptions and model architecture are quite different from those of a complementary set of models, which take the view that grid-cell activity is derived from place cells (Kropff and Treves, 2008; Dordek et al., 2016; Stachenfeld et al., 2017). Our assumptions also contrast with a third set of models in which place-cell responses are assumed to emerge largely from locally recurrent weights within hippocampus (Tsodyks et al., 1996; Samsonovich and McNaughton, 1997; Battista and Monasson, 2020; Battaglia and Treves, 1998). One challenge for those models is in explaining how to generate stable place fields through velocity integration across multiple large environments: the capacity (number of fixed points) of many fully connected neural integrator models in the style of Hopfield networks tends to be small – scaling as $\sim N$ states with $N$ neurons (Amit et al., 1985; Gardner, 1988; Abu-Mostafa and Jacques, 1985; Sompolinsky and Kanter, 1986; Samsonovich and McNaughton, 1997; Battaglia and Treves, 1998; Battista and Monasson, 2020; Monasson and Rosay, 2013) because of the absence of modular structures (Fiete et al., 2014; Sreenivasan and Fiete, 2011; Chaudhuri and Fiete, 2019; Mosheiff and Burak, 2019). There are at least two reasons why a capacity roughly equal to the number of place cells might be too small, even though the number of hippocampal cells is large: (1) a capacity equal to the number of place cells would be quickly saturated if used to tile 2D spaces: 10⁶ states from 10⁶ cells supply 10³ states per dimension. Assuming conservatively a spatial resolution of 10 cm per state, this means no more than 100 m of coding capacity per linear dimension, with no excess coding states for error correction (Fiete et al., 2008; Sreenivasan and Fiete, 2011). (2) The hippocampus sits atop all sensory processing cortical hierarchies and is believed to play a key role in episodic memory in addition to spatial representation and memory. The number of potential cortical coding states is vastly larger than the number of place cells, suggesting that the number of hippocampal coding states should grow more rapidly than linearly in the number of neurons, which is possible with our grid-driven model but not with nonmodular Hopfield-like network models with pairwise weights between neurons.

Even if our assumption that place cells primarily derive their responses from grid-like inputs combined with external cue-derived nongrid inputs is correct, place cells may nevertheless deviate from our simple perceptron model if the place response involves additional layers of nonlinear processing. There are many ways in which this can happen: place cells are likely not entirely independent of each other, interacting through population-level competition and other recurrent interactions. Dendritic nonlinearities in place cells act as a hidden layer between grid-cell input and place cell firing (Poirazi and Mel, 2001; Polsky et al., 2004; Larkum et al., 2007; Spruston, 2008; Larkum et al., 2009; Harnett et al., 2012; Harnett et al., 2013; Stuart et al., 2016). Or, if we identify our model place cells as residing in CA1, then CA3 would serve as an intermediate and locally recurrent processing layer. In principle, hidden layers that generated a one-hot encoding for space from the grid-like inputs and then drove place cells as perceptrons would make all place field arrangements realizable. However, such an encoding would require a very large number of hidden units (equal to the full range of the grid code, while the grid code itself requires only the logarithm of this number). Additionally, place cells may exhibit richer input-output transformations than a simple pointwise nonlinearity, for instance, through cellular temporal dynamics including adaptation or persistent firing. Finding ways to include these effects in the analysis of place field arrangements is a promising and important direction for future study.

In sum, combining modular grid-like inputs produces a rich spatial scaffold of place fields, on which to associate external cues, much larger than possible with nonmodular recurrent dynamics within hippocampus. Nevertheless, the allowed states are strongly constrained by the geometry of the grid-cell drive. Further, our results suggest either high volatility in the place scaffold if grid-to-place-cell weights exhibit synaptic plasticity, or suggest the possibility that grid-to-place-cell weights might be random and fixed.

In this Appendix, we introduce the geometrical framework for the study of place cells modeled as perceptrons reading out the activity of grid cells. First, we define the space of grid-like inputs via symmetry considerations and without considering explicitly their relation to spatial locations. Second, we discuss linearly separable dichotomies in the space of grid-like inputs, whose geometric arrangements are not in general position. Third, we show that the geometry of grid-like inputs is that of a polytope that can be decomposed as an orthogonal product of simplices.

We model grid-cell activity via $\{0,1\}$ spatial patterns ${\displaystyle \mathit{r}}$ that take value 1 whenever the cell is active and take value 0 otherwise (Fyhn et al., 2004; Fiete et al., 2008). To model the periodic spatial response of grid cells, we assume that the activity pattern of a grid cell defines a periodic lattice with integer period $\lambda$. For simplicity, we consider 1D model for which the spatial patterns ${\displaystyle \mathit{r}}$ are $\lambda$-periodic vectors and for which the set of activity patterns is given by the lattices $i+\lambda \mathbb{Z}$, $1\le i\le L$. We refer to the index $i$ as the phase index of the grid-cell spatial pattern. Our key results will generalize to lattices of arbitrary dimension $n$, for which the set of spatial patterns is given by the hypercube lattices ${\displaystyle \mathit{i}+{\left(\lambda \mathbb{Z}\right)}^n}$, with phase indices ${\displaystyle \mathit{i}}$ in ${\{1,\mathrm{\dots },\lambda \}}^n$.

Within a population, grid cells can have distinct periods and arbitrary phases. To model this heterogeneity, we consider a population of grid cells with $M$ possible integer spatial periods ${\displaystyle \mathit{\lambda }=({\lambda }_1,\dots ,{\lambda }_M)}$, thereby defining $M$ modules of grid cells. We assume that each module comprises all possible grid-cell-activity patterns, that is, ${\lambda }_m$ grid cells labeled by the phase indices $i$, $1\le i\le {\lambda }_m$. For convenience, we index each cell by its module index $m$ and its phase index $i$, $1\le i\le {\lambda }_m$, so that the actual component index of cell $(m,i)$, $1\le i\le {\lambda }_m$, is ${\displaystyle \sum _{n\S lt;m}{\lambda }_m+i}$. By construction of our model, at every spatial position, each module has a single active cell. Thus, at each spatial position, the grid-like input is specified by $\{0,1\}$ column vectors ${\displaystyle {\mathit{c}}_{\mathit{\lambda }}}$ of dimension $N={\sum }_{m=1}^M{\lambda }_m$, the total number of grid cells.

In principle, the inputs to place cells are defined as spatial locations. Here, by contrast, we consider grid-like inputs as the inputs to place cells, without requiring these patterns to be spatial encodings. This approach is mathematically convenient as it allows us to exploit the many symmetries of the set of grid-like inputs denoted by ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$. The set ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ contains as many grid-like inputs ${\displaystyle \mathit{c}}$ as there are choices of phase indices in each module, that is, $\mathrm{\Lambda }={\prod }_{m=1}^M{\lambda }_m$:

Here follow two examples of grid-like inputs ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ enumerated in lexicographical order for ${\displaystyle \mathit{\lambda }=(2,3)}$ and ${\displaystyle \mathit{\lambda }=(2,2,2)}$.

Observe that, albeit inspired by the spatial activity of grid cells, the set of patterns ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ has broader relevance than suggested by its use for modeling grid-like inputs. In fact, the set of patterns ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ describes any modular winner-take-all activity, whereby cells are pooled in modules with only one cell active at a time – the winner of the module.

In the following, we consider that linear read-outs of grid-like inputs determine the activity of downstream cells, called place cells (O’Keefe and Dostrovsky, 1971). The set of these linear read-outs is the vector space ${\displaystyle V_{\mathit{\lambda }}}$ spanned by the grid-like inputs ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$. The dimension of the vector space ${\displaystyle V_{\mathit{\lambda }}}$ specifies the dimensionality of the grid code. The following proposition characterizes ${\displaystyle V_{\mathit{\lambda }}}$ and shows that its dimension is simply related to the periods ${\displaystyle \mathit{\lambda }}$.

The set of grid-like inputs ${\displaystyle {\mathit{C}}_{\mathit{\lambda }}}$ specified by $M$ grid modules with integer periods ${\displaystyle \mathit{\lambda }=({\lambda }_1,\dots ,{\lambda }_M)}$ span the vector space

In particular, the embedding dimension of the grid code is ${\displaystyle \dim V_{\mathit{\lambda }}=\sum _{m=1}^M{\lambda }_m-M+1}$.

Proof. Let us denote by ${\displaystyle A_{\mathit{\lambda }}}$ a matrix formed by collecting all the column vectors from ${\displaystyle {\mathit{C}}_{\mathit{\lambda }}}$. The vector space ${\displaystyle V_{\mathit{\lambda }}}$ is the range of the matrix ${\displaystyle A_{\mathit{\lambda }}}$, which is also the orthogonal complement of ${\displaystyle \ker A_{\mathit{\lambda }}^T}$. A vector ${\displaystyle \mathit{x}=(x_{1,1},\dots ,x_{1,{\lambda }_1}|\dots \dots |x_{M,1},\dots ,x_{M,{\lambda }_M})}$ in ${ℝ}^{{\lambda }_1}\times \mathrm{\dots }\times {ℝ}^{{\lambda }_M}$ belongs to ${\displaystyle \ker A_{\mathit{\lambda }}^T}$ if and only if ${\displaystyle {\mathit{x}}^T{A}_{\mathit{\lambda }}=0}$. By construction of the matrix ${\displaystyle A_{\mathit{\lambda }}}$:

where i_m refers to the index of the active cell in module $m$. The latter characterization implies that

In turn, a vector ${\displaystyle \mathit{y}=(y_{1,1},\dots ,y_{1,{\lambda }_1}|\dots \dots |y_{M,1},\dots ,y_{M,{\lambda }_M})}$ of the orthogonal complement of ${\displaystyle \ker A_{\mathit{\lambda }}^T}$, that is, in the range of ${\displaystyle A_{\mathit{\lambda }}}$, is determined by ${\displaystyle {\mathit{x}}^T\mathit{y}=0}$ for all ${\displaystyle \mathit{x}}$ in ${\displaystyle \ker A_{\mathit{\lambda }}^T}$. From the above characterization of ${\displaystyle \ker A_{\mathit{\lambda }}^T}$, this means that ${\displaystyle \mathit{y}}$ is in the range of ${\displaystyle A_{\mathit{\lambda }}}$, that is, in ${\displaystyle V_{\mathit{\lambda }}}$, if and only if for all $a_1,\mathrm{\dots },a_M$ such that ${\sum }_{m=1}^M{a}_m=0$, we have

Substituting $a_M=-{\sum }_{m=0}^{M-1}{a}_m$ in the above relation, we have that for all $a_1,\mathrm{\dots },a_{M-1}$ in ${ℝ}^{M-1}$,

which is equivalent to ${\sum }_{i=0}^{{\lambda }_m-1}{y}_{m,i}={\sum }_{i=0}^{{\lambda }_M-1}{y}_{M,i}$ for all $m$, ${\displaystyle 1\le m\S lt;M}$. The above relation entirely specifies the range of the activity matrix ${\displaystyle A_{\mathit{\lambda }}}$, that is, ${\displaystyle V_{\mathit{\lambda }}}$, as a vector space of dimension ${\sum }_{m=1}^M{\lambda }_m-M+1$.

Every realizable multi-field structure can be implemented by a perceptron with $\mathrm{(}i\mathrm{)}$ non-negative weights, or $\mathrm{(}i\mathit{}i\mathrm{)}$ with zero threshold.

Proof. $(i)$ If $M$ is the total number of modules and 1 is the $N$-dimensional column vectors of 1, for all grid-like inputs ${\displaystyle \mathit{c}}$ in ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ we have ${\displaystyle {\mathbf{1}}^T\mathit{c}=(1,\dots ,1)\mathit{c}=M}$. Thus, for all perceptron ${\displaystyle (\mathit{w},\theta )}$ and for all real µ, we have

where $p$ is the place-cell-activity level for grid-cell pattern ${\displaystyle \mathit{c}}$ in ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$. Consequently, setting $\mu \ge {\max }_{1\le i\le N}|w_i|$, ${\displaystyle {\mathit{w}}^{\mathrm{\prime }}=\mathit{w}+\mu \mathbf{1}}$ and ${\theta }^{\prime }=\theta +\mu M$ defines a new perceptron ${\displaystyle ({\mathit{w}}^{\mathrm{\prime }},{\theta }^{\mathrm{\prime }})}$ with non-negative weights, which operates the same classification as the perceptron ${\displaystyle (\mathit{w},\theta )}$ is equivalent to ${\displaystyle p>\theta }$ The result directly follows from a similar argument by observing that for all grid-populations pattern ${\displaystyle \mathit{c}}$ in${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$

which implies that if the perceptron models ${\displaystyle (\mathit{w},\theta )}$ and ${\displaystyle (\mathit{w}-\theta \mathbf{1},0)}$ achieve the same linear classification.

Our goal is to study the multi-field structure of place-cell perceptrons, which amounts to characterize the two-class linear classifications of grid-like inputs ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$. The study of linear binary classifications has a long history in machine learning. Given a collection of $\mathrm{\Lambda }$ input patterns, there are $2^{\mathrm{\Lambda }}$ possible assignments of binary labels to these patterns, also referred to as dichotomies. In general, not all dichotomies can be linearly classified by a perceptron. Those dichotomies that can be classified are called linearly separable. An important question is to compute the number of linearly separable dichotomies, which depends on the geometrical arrangement of the inputs presented to the perceptron. Remarkably, Cover’s function counting theorem specifies the exact number of linearly separable dichotomies for $P$ inputs represented as points in a $N$-dimensional space (Cover, 1965). For inputs in general position, the number of dichotomies realizable by a zero-threshold perceptron is given by

which shows that all dichotomies are possible as long as $P\le N$. A collection of points ${\displaystyle \{{\mathit{x}}_1,\dots ,{\mathit{x}}_P\}}$ in an $N$-dimensional space is in general position if no subset of $n+1$ points lies on a $(n-1)$-dimensional plane for all $n\le N$. In our modeling framework, the inputs are collections of points representing grid-like inputs ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$. As opposed to Cover’s theorem assumptions, these grid-like inputs are not in general position as soon as we consider grid code with more than one module. For instance, it is not hard to see that for ${\displaystyle \mathit{\lambda }=(2,3)}$, the patterns $(1,0|1,0,0)$, $(1,0|0,1,0)$, ${\displaystyle (0,1|1,0,0)}$ and ${\displaystyle (0,1|0,1,0)}$ are not in general position for being the vertices of a square, therefore lying in a 2D plane. Nongeneric arrangements of grid-like inputs are due to symmetries that are inherent to the modular structure of the grid code. We expect such symmetries to heavily restrict the set of linearly separable dichotomies, therefore constraining the multi-field structure of a place cell perceptron.

We justify the above expectation by discussing the problem of linear separability for two codes that are related to the grid code. These two codes are the ‘one-hot’ code, whereby a single cell is active for all input pattern, and the ‘binary’ code, whereby the set of input patterns enumerate all possible binary vectors of activity. Exemplars of grid-like inputs for the one-hot code and the binary code are given for $N=3$ input cells by

From a geometrical point of view, a set of points representing the grid-like inputs ${\displaystyle {\mathcal{S}}_J\subset \mathcal{C}}$ is linearly separable if there is a hyperplane separating the points ${\displaystyle \mathcal{S}}$ from the other points ${\displaystyle \mathcal{C}\setminus \mathcal{S}}$. The existence of a hyperplane separating a single point from all other points is straightforward when the set of patterns correspond to the vertices of a convex polytope. Then, every vertex can be linearly separated from the other points for being an extreme point. It turns out that both the population patterns of the one-hot code and of the binary code represent the vertices of a convex polytope: a simplex for the single-cell code and a hypercube for the binary code. However, because these vertices are in general position for the single-cell code but not for the binary code, the fraction of linearly separable dichotomies drastically differs for the two codes.

Let us first consider the $N$ points whose coordinates are given by $C_{\mathrm{oh}}$. The convex hull of $C_{\mathrm{oh}}$ is the canonical $(N-1)$-dimensional simplex. Thus, any sets of $k$ vertices, $1\le k\le N$, specifies a $(k-1)$-dimensional face of the simplex, and as such, is a linearly separable $k$-dichotomy. This immediately shows that all dichotomies are linearly separable. This result follows from the fact that the $N$ points in $C_{\mathrm{oh}}$ are in general position. Let us then consider the $2^N$ points whose coordinates are given by $C_{\mathrm{b}}$. The convex hull of $C_{\mathrm{b}}$ is the canonical $N$-dimensional hypercube. Thus, by contrast with ${\displaystyle C_{\mathrm{o}\mathrm{h}}}$, the points in $C_{\mathrm{b}}$ are not in general position. As a result, there are dichotomies that are not linearly separable as shown by considering. For instance, the pair $\{(1,0,0)$, $(0,1,0)\}$ and the pair $\{(0,0,0)$, $(1,1,1)\}$ can be linearly separated from the other points of the hypercube. Determining the number of linearly separable sets of hypercube vertices is a hard combinatorial problem that has attracted a lot of interest (Peled and Simeone, 1985; Hegedüs and Megiddo, 1996). Unfortunately, there is no efficient characterization of that number as a function of the dimension $N$. However, it is known that out of the $2^{2^N}$ possible dichotomies, the total number of linearly separable dichotomies scales as $2^{N^2}$ in the limit of large dimension $N\to \mathrm{\infty }$ (Irmatov, 1993). This shows that only a vanishingly small fraction of hypercube dichotomies are also linearly separable.

For integer periods ${\displaystyle \mathit{\lambda }=({\lambda }_1,\dots ,{\lambda }_M)}$, the convex hull generated by ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$, the set of grid-cell-population patterns, determines a $d$-dimensional polytope ${\displaystyle H_{\mathit{\lambda }}}$, with $d\mathrm{=}{\mathrm{\sum }}_{m\mathrm{=}\mathrm{1}}^M{\lambda }_m\mathrm{-}M$, defined as ${\displaystyle H_{\mathit{\lambda }}={\mathrm{\Delta }}^{{\lambda }_1}\times \dots \times {\mathrm{\Delta }}^{{\lambda }_M}}$ where ${\mathrm{\Delta }}^{{\lambda }_m}$, $\mathrm{1}\mathrm{\le }m\mathrm{\le }M$, denotes the $\mathrm{(}{\lambda }_m\mathrm{-}\mathrm{1}\mathrm{)}$-simplex specified by the ${\lambda }_m$ points: $\mathrm{(}\mathrm{1}\mathrm{,}\mathrm{0}\mathrm{,}\mathrm{\dots }\mathrm{,}\mathrm{0}\mathrm{)}\mathrm{,}\mathrm{(}\mathrm{0}\mathrm{,}\mathrm{1}\mathrm{,}\mathrm{0}\mathit{}\mathrm{\dots }\mathrm{,}\mathrm{0}\mathrm{)}\mathrm{,}\mathrm{\dots }\mathrm{,}\mathrm{(}\mathrm{0}\mathrm{,}\mathrm{\dots }\mathrm{,}\mathrm{0}\mathrm{,}\mathrm{1}\mathrm{)}$.

Before proving the product decomposition of ${\displaystyle H_{\mathit{\lambda }}}$, let us make a couple of observations. First, observe that all the vectors ${\displaystyle \mathit{c}}$ in ${\displaystyle C_{\mathit{\lambda }}}$ satisfies ${\displaystyle {\mathbf{1}}^T\mathit{c}=M}$, so that all edges ${\displaystyle \mathit{c}-{\mathit{c}}^{\mathrm{\prime }}}$, with ${\displaystyle \mathit{c}}$, ${\displaystyle {\mathit{c}}^{\mathrm{\prime }}}$ in ${\displaystyle C_{\mathit{\lambda }}}$, lie in the same hyperplane of the vector space ${\displaystyle V_{\mathit{\lambda }}}$. By Proposition 1, ${\displaystyle V_{\mathit{\lambda }}}$ has dimension $N={\sum }_m{\lambda }_m-M+1$, this implies that the dimension of the polytope ${\displaystyle H_{\mathit{\lambda }}}$ is at most $d=N-1$. Second, observe that the set ${\displaystyle C_{\mathit{\lambda }}}$ is left unchanged by the symmetry operators $J_{{\lambda }_m}$, $1\le m\le M$, where $J_{{\lambda }_m}$ cyclically shifts downward the mth module coordinates of the vectors in ${\displaystyle C_{\mathit{\lambda }}}$. The operators $J_{{\lambda }_m}$ admit the matrix representation

where $I_n$ denotes the identity matrix in ${ℝ}^n\times {ℝ}^n$. Notice that the matrices $J_{{\lambda }_m}$ satisfy $J_{{\lambda }_m}^T{J}_{{\lambda }_m}=I_{{\lambda }_1+\mathrm{\dots }+{\lambda }_M}$ showing that the operators $J_{{\lambda }_m}$ are isometries in ${\displaystyle {\mathit{V}}_{\mathit{\lambda }}}$. Moreover, observe that for all ${\displaystyle \mathit{c}}$, ${\displaystyle {\mathit{c}}^{\mathrm{\prime }}}$ in ${\displaystyle C_{\mathit{\lambda }}}$, there are integers of $k_1,\mathrm{\dots },k_M$ such that ${\displaystyle J_{{\lambda }_1}^{k_1}\dots J_{{\lambda }_1}^{k_M}\mathit{c}={\mathit{c}}^{\mathrm{\prime }}}$. This shows that each vector in ${\displaystyle C_{\mathit{\lambda }}}$ plays the same role in defining the geometry of ${\displaystyle H_{\mathit{\lambda }}}$, and thus ${\displaystyle H_{\mathit{\lambda }}}$ is vertex-transitive. In particular, every vector in ${\displaystyle C_{\mathit{\lambda }}}$ represents an extreme point of the convex hull ${\displaystyle H_{\mathit{\lambda }}}$. As a result, ${\displaystyle H_{\mathit{\lambda }}}$ is a polytope with as many vertices as the cardinality of ${\displaystyle C_{\mathit{\lambda }}}$, that is, $\mathrm{\Lambda }={\prod }_{m=1}^M{\lambda }_m$. The product decomposition of the polytope ${\displaystyle H_{\mathit{\lambda }}}$ then follows from a simple recurrence argument over the number of modules $M$.

Appendix 1—figure 1

Download asset Open asset

Proof. In order to relate the geometrical structure of ${\displaystyle H_{\mathit{\lambda }}}$ to that of simplices, let us introduce ${\displaystyle {\mathit{e}}_i}$, $1\le i\le {\lambda }_M$, the elementary unit vector corresponding to the $i$-th coordinate of ${ℝ}^{{\lambda }_M}$. The set ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ has the following product structure

where ${\displaystyle {\mathcal{C}}_{{\mathit{\lambda }}^{\mathrm{\prime }}}}$ is the set of vectors for $M-1$ modules with periods ${\displaystyle {\mathit{\lambda }}^{\mathrm{\prime }}=\{{\lambda }_1,\dots ,{\lambda }_{M-1}\}}$. The product structure of the set ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ transfers to the convex hull $H_{\lambda }$ it generates. Specifically, we have

where we have recognized that the convex hull of the set of elementary basis vectors ${\displaystyle {\mathit{e}}_i}$, $1\le i\le {\lambda }_M$, is precisely the canonical $({\lambda }_M-1)$-simplex. Thus, we have shown that ${\displaystyle H_{\mathit{\lambda }}=H_{{\mathit{\lambda }}^{\mathrm{\prime }}}\times {\mathrm{\Delta }}^{{\lambda }_M}}$. Proceeding by recurrence on the number of modules, one obtains the announced decomposition of the convex hull as a product $H={\mathrm{\Delta }}^{{\lambda }_1}\times \mathrm{\dots }\times {\mathrm{\Delta }}^{{\lambda }_M}$, where ${\mathrm{\Delta }}^{{\lambda }_M}$, $1\le m\le M$, is the canonical $({\lambda }_m-1)$-simplex.

The above orthogonal decomposition suggests that the problem of determining the linearly separable dichotomies of grid-like inputs is related to that of determining the linearly separable Boolean functions. Indeed, the polytope defined by grid-like inputs with $M$ modules contains $M$-dimensional hypercubes, for which many dichotomies are not linearly separable. As counting the linearly separable Boolean functions is a notoriously hard combinatorial problem, it is unlikely that one can find a general characterization of the linearly separable dichotomies of grid-like inputs. However, it is possible to give some explicit results for the case of two modules $M$ or for the case of $k$-dichotomies for small cardinality $k$.

In this Appendix, we establish combinatorial results about the properties and the cardinality of linearly separable dichotomies of grid-like inputs. First, we show that linearly separable dichotomies can be partitioned in classes, each indexed by a combinatorial object called Young diagram. Second, we exploit related combinatorial objects, called Young tableaux, to show that not all Young diagrams correspond to linearly separable dichotomies. Third, we utilize Young diagrams to characterize dichotomies for which one class of labeled patterns has small cardinality ${\displaystyle k=1,\dots ,4}$. Fourth, we count the exact number of linearly separable dichotomies for grid-like inputs with two modules.

To count linearly separable dichotomies, we first show that these dichotomies can be partitioned in classes that are indexed by Young diagrams. Young diagrams are useful combinatorial objects that have been used to study, e.g., the properties of the group representations of the symmetric group and of the general linear group. Young diagrams are formally defined as follows:

A d-dimensional Young diagram is a subset D of lattice points in the positiveorthant of a d-dimensional integral lattice, which satisfies the following:

1. If ${\displaystyle (n_1,\dots ,n_i,\dots ,n_d)\in D}$ and ${\displaystyle n_i\S gt;0}$, then ${\displaystyle (n_1,\dots ,n_i-1,\dots ,n_d)\in D}$.

2. For any positive integer i ≤ d, and any non negative integers, m, p, with m > p, the restriction of D to the hyperplane n_i = m is a (d−1)-dimensional Young diagram that covers the (d − 1)-dimensional Young diagram formed by the restriction of S to the hyperplane n_i = p.

Moreover, the size of the diagram D, denoted by |D|, is defined as the number of lattice points in D.

Young diagrams have been primarily studied for d = 2 because their use allows oneto conveniently enumerate the partitions of the integers. For d = 2, there are differentconventions for representing Young diagrams pictorially. Hereafter, we follow the Frenchnotations, where Young diagrams are left justified lattice rows, whose length decreaseswith height. For the sake of clarity, Fig. 1a depicts the 5 Young diagrams associated to thepartitions of 4: 4, 3 + 1, 2 + 2, 2 + 1 + 1 and 1 + 1 + 1 + 1: Young diagrams have been less studiedfor dimensions d ≥ 3 and only a few of their combinatorial properties are known. Fig. 1brepresents a 3-dimensional diagram, together with two 2-dimensional restrictions (red edgesfor n₃ = 1 and yellow edges for n₃ = 3). Observe that these restrictions are 2-dimensionalYoung diagrams, and that the restriction corresponding to n₃ = 1 covers the restriction corresponding to n₃ = 3. Young diagrams can equivalently be viewed as arrays of boxesrather than lattice points in the positive orthant. This corresponds to identifying each latticepoint ${\displaystyle (n_1,\dots ,n_d)\in D}$ with the unit cube ${\displaystyle (n_1-1,n_1)\times \dots \times (n_d-1,n_d)}$.

Before motivating the use of Young diagrams, let us make a few remarks about the set ofdichotomies that can be realized by a perceptron with fixed weight vector (ω, θ). First, recallthat with no loss of generality we can restrict the weight vectors ω to be nonnegative byProposition 2. Second, by permutation invariance, there is no loss of generality in consideringa perceptron (ω, θ) for which the weight vector.

is such that the weights are ordered within each module: ${\displaystyle w_{m,1}\S lt;\dots \S lt;w_{m,{⋋}_m}}$, for all ${\displaystyle m,1\le m\le M}$. We refer to weight vectors having this module-specific, increasing order propertyas being a modularly ordered weight vector. Bearing these observations in mind, the following proposition establishes the link between Young diagrams and perceptrons.

Given integer periods ${\displaystyle \mathit{\lambda }=({\lambda }_1,\dots ,{\lambda }_M)}$, for all modularly ordered, non-negative, weight vectors ${\displaystyle \mathit{w}}$ and for all thresholds $\theta$, the lattice set

is a $M$-dimensional Young diagram in $\mathrm{\{}\mathrm{1}\mathrm{,}\mathrm{\dots }\mathit{}{\lambda }_{\mathrm{1}}\mathrm{\}}\mathrm{\times }\mathrm{\dots }\mathrm{\times }\mathrm{\{}\mathrm{1}\mathrm{,}\mathrm{\dots }\mathit{}{\lambda }_M\mathrm{\}}$.

In other words, under assumption of modularly ordered, non-negative weights, the phase indices of inactive grid cells form a Young diagram.

Proof. The Young diagram properties directly follow from the ordering of weights within modules. For instance, it is easy to see that if $(i_1,\mathrm{\dots },i_M)\le (j_1,\mathrm{\dots },j_M)$ for the component-wise partial order in $\{1,\mathrm{\dots }{\lambda }_1\}\times \mathrm{\dots }\times \{1,\mathrm{\dots }{\lambda }_M\}$, then ${\displaystyle (j_1,\dots ,j_M)\in \mathcal{D}(\mathit{w},\theta )}$ implies ${\displaystyle (i_1,\dots ,i_M)\in \mathcal{D}(\mathit{w},\theta )}$. Indeed, we necessarily have

By the above proposition, given a grid code with $M$ modules, every perceptron ${\displaystyle (\mathit{w},\theta )}$ acting on that grid code can be associated to a unique $M$-dimensional Young diagram ${\displaystyle \mathcal{D}(\mathit{w},\theta )}$ after ordering the components of ${\displaystyle \mathit{w}}$ within each module. Conversely, if a $M$-dimensional Young diagram $D^{\prime }$ can be associated to a perceptron ${\displaystyle (\mathit{w},\theta )}$ with modularly ordered, non-negative weights, we say that ${\displaystyle {\mathcal{D}}^{\mathrm{\prime }}=\mathcal{D}(\mathit{w},\theta )}$ is realizable. Then a natural question to ask is: are all $M$-dimensional Young diagrams realizable by perceptrons? It turns out that perceptrons exhaustively enumerate all $M$-dimensional Young diagrams if $M\le 2$, but there are unrealizable Young diagrams as soon as ${\displaystyle M>2}$.

Understanding why there are unrealizable Young diagrams as soon as ${\displaystyle M>2}$ involves using combinatorial objects that are closely related to Young diagrams, called Young tableaux.

Given a Young diagram $\mathrm{D}$, a Young tableau $\mathrm{T}$ is obtained by labeling the lattice points – or filling in the boxes – of $\mathrm{D}$ with the integers $\mathrm{1}\mathrm{,}\mathrm{2}\mathrm{,}\mathrm{\dots }\mathrm{,}\mathrm{|}D\mathrm{|}\mathrm{,}$ such that each number occurs exactly once and such that the entries are increasing across each row (to the right) and across each column (to the top).

Here are two examples of Young tableaux that are distinct labeling of the same Young diagram:

Appendix 2—scheme 1

Download asset Open asset

Just as Young diagrams, Young tableaux are naturally associated to perceptrons. The following arguments specify the correspondence between perceptrons and Young tableaux. Given a perceptron ${\displaystyle (\mathit{w},\theta )}$ with modularly ordered, non-negative weights, let us order all patterns in ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ by increasing level of perceptron activity. Specifically, set ${\displaystyle {\mathcal{J}}_0={\mathcal{C}}_{\mathit{\lambda }}}$ and define iteratively for $k$, ${\displaystyle 0\le k\S lt;\mathrm{\Lambda }}$,

With no loss of generality, we can assume that all patterns achieve distinct levels of activity, so that there is a unique minimizer for all ${\displaystyle k}$, ${\displaystyle 0\le k<\mathrm{\Lambda }}$. With that assumption, the sequence ${\displaystyle {\mathit{c}}_k^{\star }(\mathit{w})}$, $1\le k\le \mathrm{\Lambda }$, enumerates unambiguously all patterns in ${\displaystyle {\mathcal{C}}_{\mathit{\lambda }}}$ by increasing level of activity. The Young tableau associated to the perceptron ${\displaystyle (\mathit{w},\theta )}$, denoted by ${\displaystyle \mathcal{T}(\mathit{w},\theta )}$, is then obtained by labeling lattice points of the Young diagram ${\displaystyle \mathcal{D}(\mathit{w},\theta )}$ by increasing level of activity as in the sequence ${\displaystyle {\mathit{c}}_k^{\star }(\mathit{w})}$, ${\displaystyle 1\le k\le |\mathcal{D}(\mathit{w},\theta )|}$. One can check that such labeling yields a tableau as the resulting labels increase along each rows (to the right) and columns (to the top). Within this framework, we say that a Young tableau ${\displaystyle {\mathcal{T}}^{\mathrm{\prime }}}$ is realizable if there is a perceptron ${\displaystyle (\mathit{w},\theta )}$ such that ${\displaystyle {\mathcal{T}}^{\mathrm{\prime }}=\mathcal{T}(\mathit{w},\theta )}$. Finally, let us define the sequence of thresholds ${\displaystyle {\theta }_k(\mathit{w})}$, ${\displaystyle 0\le k\le \mathrm{\Lambda }+1}$, such that ${\theta }_0=-\mathrm{\infty }$, ${\displaystyle {\theta }_{\mathrm{\Lambda }+1}(\mathit{w})=\mathrm{\infty }}$, and for${\displaystyle 0\S lt;k\le \mathrm{\Lambda }}$

Then, observe that for all $k$, $0\le k\le \mathrm{\Lambda }$, the set of active patterns ${\displaystyle {\mathcal{J}}_k(\mathit{w})}$ is linearly separable for threshold $\theta$ satisfying ${\displaystyle {\theta }_k(\mathit{w})\le \theta <{\theta }_{k+1}(\mathit{w})}$. In fact, the sequence ${\displaystyle \{{\mathcal{J}}_k(\mathit{w}){\}}_{0\le k\le \mathrm{\Lambda }}}$ represents all the linearly separable dichotomies realizable by changing the threshold of a perceptron with weight vector ${\displaystyle \mathit{w}}$. This fact will be useful to prove the following proposition, which justifies considering Young tableaux.

All $M$-dimensional Young diagrams are realizable if and only if all $\mathrm{(}M\mathrm{-}\mathrm{1}\mathrm{)}$-dimensional Young tableaux are realizable.

Observe that the above proposition does not mention the periods ${\lambda }_1,\mathrm{\dots },{\lambda }_M$. This is because the proposition deals with the correspondence between $m$-dimensional Young diagrams and $(M-1)$-dimensional Young tableaux for all possible assignments of periods.

Proof. In this proof, we use prime notations for quantities relating to $M-1$ modules and regular notations for quantities relating to $m$ modules. For instance, ${\displaystyle \mathit{\lambda }}$ denotes an arbitrary assignment of $m$ periods $\{{\lambda }_1,\mathrm{\dots },{\lambda }_M\}$ and ${\displaystyle {\mathit{\lambda }}^{\mathrm{\prime }}}$ denotes its $m-1$ first components $\{{\lambda }_1,\mathrm{\dots },{\lambda }_{M-1}\}$. With this preamble, we give the ‘if’ part of proof in $(i)$ and the ‘only if’ part in $(ii)$.

(i) Given a $(M-1)$-dimensional Young tableau ${\displaystyle {\mathcal{T}}^{\mathrm{\prime }}}$ with diagram ${\displaystyle {\mathcal{D}}^{\mathrm{\prime }}}$, let us consider the smallest periods ${\displaystyle {\mathit{\lambda }}^{\mathrm{\prime }}}$ such that ${\displaystyle {\mathcal{D}}^{\mathrm{\prime }}\subset \{1,\dots ,{\lambda }_1\}\times \dots \times \{1,\dots ,{\lambda }_{M-1}\}}$. The ‘if’ part of the proof will follow from showing that if all $(M-1)$-dimensional tableaux ${\displaystyle {\mathcal{T}}^{\mathrm{\prime }}}$ with Young diagram ${\displaystyle {\mathcal{D}}^{\mathrm{\prime }}}$ are realizable, than all ${\displaystyle M}$-dimensional Young diagrams whose restriction to ${\displaystyle \{1,\dots {\lambda }_1\}\times \dots \times \{1,\dots {\lambda }_{M-1}\}\times \{1\}}$ is ${\displaystyle {\mathcal{D}}^{\mathrm{\prime }}}$ are realizable. To prove this property, observe that all the $M$-dimensional Young diagrams with restriction ${\displaystyle {\mathcal{D}}^{\mathrm{\prime }}}$ are obtained as finite sequences of ${\displaystyle (M-1)}$-dimensional Young diagrams ${\displaystyle {\mathcal{D}}^{\mathrm{\prime }}={\mathcal{D}}_1^{\mathrm{\prime }}\supset {\mathcal{D}}_2^{\mathrm{\prime }}\supset \dots \supset {\mathcal{D}}_{{\lambda }_M}^{\mathrm{\prime }}}$, for some ${\displaystyle {\lambda }_M}$ specifying the minimum period in the mth dimension. For all such sequences, consider a tableau ${\displaystyle {\mathcal{T}}^{\mathrm{\prime }}}$ labeling $D^{\prime }$ such that for all $i$, $1\le i\le {\lambda }_M-1$, the labels of $D_{i+1}^{\prime }$ are smaller than the labels $D_i^{\prime }\setminus D_{i+1}^{\prime }$. Such a tableau is always possible because of the nested property of the sequence of diagrams $D_i^{\prime }$, $1\le i\le {\lambda }_M$. Now, suppose that the Young tableau $T^{\prime }$ is realizable. This means that there is a perceptron ${\displaystyle ({\mathit{w}}^{\mathrm{\prime }},{\theta }^{\mathrm{\prime }})}$ acting on the grid-like inputs in ${\displaystyle {\mathcal{C}}_{{\mathit{\lambda }}^{\mathrm{\prime }}}}$ such that ${\displaystyle {\mathcal{T}}^{\mathrm{\prime }}=\mathcal{T}\mathbf{(}{w}^{\mathrm{\prime }},{\theta }^{\mathrm{\prime }})}$. With no loss of generality, the weight vector ${\displaystyle {\mathit{w}}^{\mathrm{\prime }}}$ specifies a sequence of patterns ${\displaystyle {\mathit{c}}_k^{\star }({\mathit{w}}^{\mathrm{\prime }})}$, $1\le k\le {\mathrm{\Lambda }}^{\prime }$, and a sequence of thresholds ${\displaystyle {\theta }_k({\mathit{w}}^{\mathrm{\prime }})}$, $1\le k\le {\mathrm{\Lambda }}^{\prime }$, such that ${\displaystyle (1)}$ enumerates the elements of ${\displaystyle {\mathcal{C}}_{{\mathit{\lambda }}^{\mathrm{\prime }}}}$ by increasing level of activity and ${\displaystyle (2)}$ for all ${\displaystyle 0\le k\le |{\mathcal{D}}^{\mathrm{\prime }}|}$, the set of active patterns ${\displaystyle {\mathcal{J}}_k(\mathit{w})}$ defined in (29) is linearly separable if and only if ${\displaystyle {\theta }_k({\mathit{w}}^{\mathrm{\prime }})\le \theta <{\theta }_{k+1}({\mathit{w}}^{\mathrm{\prime }})}$. Then by construction, the diagrams $D_i^{\prime }$, $1\le i\le {\lambda }_M$, are realized by a perceptron ${\displaystyle ({\mathit{w}}^{\mathrm{\prime }},{\theta }_i^{\mathrm{\prime }})}$, where every ${\theta }_i^{\prime }\ge {\theta }^{\prime }$ is such that ${\displaystyle {\theta }_{\mathrm{\Lambda }-|{\mathcal{D}}_i^{\mathrm{\prime }}|}({\mathit{w}}^{\mathrm{\prime }})<{\theta }_i^{\mathrm{\prime }}<{\theta }_{\mathrm{\Lambda }-|{\mathcal{D}}_i^{\mathrm{\prime }}|+1}({\mathit{w}}^{\mathrm{\prime }})}$. We are now in a position to construct a $M$-module perceptron ${\displaystyle (\mathit{w},{\theta }^{\mathrm{\prime }})}$ realizing the sequence $D^{\prime }=D_1^{\prime }\supset D_2^{\prime }\supset \mathrm{\dots }\supset D_{{\lambda }_M}^{\prime }$. To do so, it is enough to specify the components $w_{M,1},\mathrm{\dots },w_{M,{\lambda }_M}$ of the Mth module of a weight vector ${\displaystyle \mathit{w}}$ since the other components will coincide with ${\displaystyle {\mathit{w}}^{\mathrm{\prime }}}$. One can check that choosing $w_{M,i}={\theta }_i^{\prime }-{\theta }^{\prime }$ defines an admissible increasing sequence of non-negative weights.