Nonexistence of Energy Function for Surface Diffeomorphisms with Nontrivial Basic Sets of Serpentine Geometric Type

The Lyapunov function is a powerful tool for the study of dynamical systems. In particular, the existence of such a function for any dynamical system, established by C. Conley in 1978, underlies the proof of the criterion of Ω-stability for dynamical systems [1]. The decreasing property of the Lyapunov function outside the chain-recurrent set makes it possible to identify attracting neighborhoods of the attractors of a dynamical system as level sets of the function. If, in addition, the set of critical points of the Lyapunov function coincides with the chain-recurrent set of the system (such a function is called an energy function ), then this function

• ^# The research was carried out within the framework of the Fundamental Research Program of the National Research University Higher School of Economics (Project № HSE-BR-2025-81).

• (0 2026 Barinova, М. K. and Tirskaya, К. Yu.

is qualitatively consistent with the dynamics of the system. For continuous-time systems modeling dissipative processes, such a function naturally arises as an energy function. The first investigations into the construction of an energy function, i.e., a smooth Lyapunov function whose set of critical points coincides with the chain-recurrent set of the system, were carried out by S. Smale. In 1961, in [2], he established the existence of a Morse energy function for gradient-like flows. Later, in 1968, K. Meyer [3] generalized this result by constructing a Morse–Bott energy function for an arbitrary Morse–Smale flow. In 1985, J. Franks [4] observed that the combination of the results of [1] and [5] proves that every flow defined on a compact manifold admits an energy function.

For discrete dynamical systems the situation is substantially more complicated. The first examples of cascades without energy functions were constructed on three-dimensional manifolds by D. Pixton in 1977 [6]. These were gradient-like diffeomorphisms, where the obstacle to the existence of an energy function was the wild embedding of saddle separatrices into the supporting manifold. Moreover, in the same work, D. Pixton proved the existence of a Morse energy function for any Morse–Smale diffeomorphism on a surface.

Since such obstacles are absent in regular systems on surfaces, all of them admit a Morse energy function. For systems with chaotic dynamics on surfaces, the existence of an energy function has also been established in cases where the diffeomorphism is Q-stable and the topological dimension of its nontrivial basic sets is greater than or equal to one.

However, in the case where the chain-recurrent set of a diffeomorphism contains at least one zero-dimensional nontrivial basic set, the question of the existence of an energy function remains open.

The first results on the classification of zero-dimensional basic sets belong to V. Z. Grines and H. H. Kalai. In their work [7], they obtained a topological classification of such sets without pairs of conjugated points.

A complete classification of nontrivial zero-dimensional basic sets was given by C. Bonatti and R. Langevin in 1998 [8]. The authors proved that the topological conjugacy class of the restriction of a diffeomorphism to the canonical region is characterized by the geometric type of a Markov partition of the hyperbolic set: this is simple combinatorial information describing in which order, at which place, and in which direction the image of a certain rectangle of the Markov partition intersects the original rectangles. Thus, all nontrivial zero-dimensional basic sets of a diffeomorphism can be divided into classes according to the geometric type of the Markov partition, which describes the dynamics of this diffeomorphism. Later, in 2002, F. Beguin described which types are realizable and which are not. The realizability of a geometric type means the existence of an orientation-preserving, structurally stable diffeomorphism ¹ , whose saddle basic set admits a Markov partition of this geometric type [9].

Returning to the question of the existence of an energy function, let us recall that at present it is known that Q-stable 2-diffeomorphisms without zero-dimensional nontrivial basic sets admit such a function [6, 10, 11]. In 2022, it was proved in [12] that if a 2-diffeomorphism has at least one zero-dimensional basic set without pairs of conjugated points, then it does not admit an energy function. Until recently, there were no results concerning the existence of an energy function for diffeomorphisms with a zero-dimensional basic set with pairs of conjugated points . One of the classical examples of such sets is the Smale horseshoe. In 2024, M. K. Barinova and K. Y. Tirskaya [13] proved that if one of the basic sets of an Q-stable diffeomorphism f : M ² ^ M ² is a Smale horseshoe on a disk, then the diffeomorphism f does not admit an energy function.

In this paper, we consider zero-dimensional nontrivial saddle basic sets, each of which admits a Markov partition consisting of a single rectangle R intersecting its image f (R) as shown in Fig. 1. We will say that such basic sets admit a Markov partition of the serpentine geometric type .

Fig. 1. Markov partition of serpentine geometric type.

We will call such sets basic sets of serpentine type . It will be shown below that they contain pairs of conjugated points, and therefore their study extends the class of diffeomorphisms for which the question of the existence of an energy function has been resolved.

The main result of this paper is the following theorem:

Theorem 1. Let f : M ² ^ M ² be an orientation-preserving, structurally stable dif-feomorphism defined on a closed, smooth, orientable manifold M ² . Suppose that f has a nontrivial zero-dimensional saddle basic set that admits a Markov partition of serpentine geometric type, and that the set R U f (R) lies on a disk. Then f does not admit an energy function.

2.    Basic Definitions

We define the class of dynamical systems, with which we will work. We consider orientation-preserving, structurally stable diffeomorphisms f : M ⁿ ^ M ⁿ , where M ⁿ is a closed, smooth, orientable n-manifold. From [14–17] it follows that structural stability of the system leads to the existence of a hyperbolic structure on the chain-recurrent set R f . In this case, the set R f for a diffeomorphism from our systems’ class coincides with the nonwandering set NW(f ), and the periodic orbits are dense in R f [18, Theorem 4.1]. Moreover, the Smale spectral decomposition theorem [19] holds, namely: NW(f ) can be represented as a disjoint union of finitely many subsets, each of which is compact, invariant, and topologically transitive. Such subsets are called the basic sets of the diffeomorphism f . If a basic set is an isolated periodic orbit, it is called trivial ; otherwise, it is called nontrivial . Thus, NW(f ) = Л 1 U Л 2 U • • • U Л _s , where Лi, i = 1 ,..., s , are the basic sets.

Dynamical hyperbolicity is expressed in the fact that every point p G NW(f ) possesses unstable and stable manifolds:

Wps = |x G Mn : d ff k(p), fk(x)^ ^ 0, when k ^ +wj and

Wp = Xх G Mn : d ffk(p),fk(x^ ^ 0, when k ^ -wj , which are the images of Rq and Rn-q, 0 C q C n, under injective immersions, where d is a Riemannian metric on the manifold. The images of the q-dimensional ball Bq and the (n — q)-dimensional ball Bεnu-q , whose centers lie at the origin and whose radii are equal to ε, under these immersions will be denoted by Wlsoc and Wluoc , respectively.

Let Л be a zero-dimensional nontrivial basic set of a 2-diffeomorphism from the class under consideration. Then dim W p = dim W £ = 1, V x G Л. Denote by (x,y) ^s (resp., [ x,y ] ^s ) the open (resp., closed) arc of the stable manifold W _x ^s bounded by the points x, y ∈ W _x ^s , and by ( x,y ) ^u (resp., [ x,y ] ^u ) the open (resp., closed) arc of the unstable manifold W“ bounded by the points x, y ∈ W _x ^u .

Definition 1. Points x,y G Л are called a pair of conjugated points if Wp = W s , W“ = W y¹ , and moreover ( x, y) ^s 0 Л = 0, ( x, y) ^u 0 Л = 0 (see Fig. 2).

Fig. 2. Pair of conjugated points x and y .

In the general case, a Lyapunov function is defined via the chain-recurrent set R f , but for the class of dynamical systems under consideration it can be introduced as follows:

Definition 2. A Lyapunov function for a structurally stable diffeomorphism f : M ² ^ M ² is a continuous function ^ : M ² ^ R with the following properties:

• 1.    if x G NW(f ), then ^ ( f ( x )) < ^ ( x );

• 2.    V x,y G NW(f ), ^ ( x ) = ^ ( y ) ^ x,y belong to the same basic set.

Definition 3. A point p G M ² is called regular for a continuous function ^ : M ² ^ R if there exists a neighborhood V _p ⊂ M ² of the point p and a homeomorphism onto its image ф р : V p ^ R ² , ф р (у) = (xi(y),x2(y)), such that x i (p) =0, i G { 1 , 2 } , and ^ ( y ) = ^ ( p ) + x ₂ (y). Otherwise, the point is called critical .

Thus, in local charts the level lines of the function ^ (sets such as ^ ( y ) = c ) take the form ^ ( p ) + x 2 ( y ) = c , and hence they are horizontal lines x 2 ( y ) = const.

Definition 4. Lyapunov function ϕ is called an energy function if the set of its critical points coincides with NW(f ).

Note that if an energy function ϕ is smooth, then at each of its critical points x the gradient grad(^(x)) = 0. According to the results of [20], the following statement about an energy function holds:

Statement 1. If ϕ is an energy function for a structurally stable diffeomorphism f , then for all x G W p (respectively, for all x G W pU ) the inequality ^ ( x ) ^ ^ ( p ) ( ^ ( x ) C ^ ( p )) holds. Moreover, if x / NW(f ), then ^ ( x ) > ^ ( p ) ( ^ ( x ) < ^ ( p )).

To describe the dynamics of a structurally stable surface diffeomorphism in a neighborhood of a saddle basic set, we will use Markov partitions. This tool will play a central role in the present work, and therefore we introduce the formal definition.

A rectangle is an embedding of the square [0,1]2 into the surface. If a rectangle R is given, then two opposite sides of R are called stable or horizontal, and their union is denoted by ∂sR; the other two sides are called unstable or vertical, and their union is denoted by ∂uR, respectively. A horizontal subrectangle of a rectangle R is a rectangle H ⊂ R such that its vertical sides are contained in both vertical sides of R. Such a subrectangle is called strict if its horizontal sides do not coincide with the horizontal sides of R. The notion of a (strict) vertical subrectangle is introduced similarly.

Consider a saddle basic set Л of a structurally stable diffeomorphism of a closed, smooth, orientable surface M ² and a rectangle R embedded into M ² . The rectangle R is said to be well positioned with respect to the basic set Л if:

• •    the boundary of R does not intersect Л;

• •    every connected component of W ^s (Л) П R is an arc intersecting R horizontally, that is, connecting one vertical side with the other;

• •    every connected component of W ^и (Л) П R is an arc intersecting R vertically (see Fig. 3);

WU(R)

R : I i :

Fig. 3. A rectangle R well positioned with respect to the basic set Л.

Definition 5. Let f be a structurally stable diffeomorphism of a closed surface M ² , and let Л be a saddle basic set for f . A Markov partition for Л is a finite sequence R = { R 1 , . . . , R _m } of pairwise disjoint rectangles embedded in M ² , well positioned with respect to Л, whose union contains Л, and which satisfy the following property:

V i,j <  m , every connected component V of the set f (R i ) П R j is simultaneously a strict vertical subrectangle of the rectangle R j and the image of some strict ( * ) horizontal subrectangle H of the rectangle R i in R j .

In [21] R. Bowen showed that every saddle basic set of a structurally stable diffeomorphism on a surface admits a Markov partition.

For problems of topological classification, C. Bonatti and R. Langevin additionally endowed a Markov partition with its geometric type [8]. Its precise definition will be given below.

Definition 6. Let f and f ‘ be two mappings with closed invariant sets Л and Л ’ , respectively. Then f and f ‘ are said to be locally topologically conjugated on Л and Л ’ if for any neighborhoods V (Л) and V ‘ (Л ‘ ) there exist their subneighborhoods U С V and U ‘ С V ‘ , as well as a homeomorphism h : U 4 U ‘ , such that h(Л) = Л ’ and the following diagram commutes:

U (Л) 4   U (Л)

↓h         ↓h u‘(Л) -4 u‘(Л‘)

that is, for every point x G U (Л) the equality h(f ( x )) = f ‘ (h(x)) holds.

Thus, the homeomorphism h maps not only the trajectories of the system f | л to the trajectories of the system f ‘ | л ‘ , but also all nearby trajectories (more precisely, the parts of trajectories belonging to the corresponding neighborhoods U and U ^′ ).

For every saddle basic set Л of a structurally stable diffeomorphism f on a surface, there exists a canonical invariant neighborhood, denoted by A(f, Л). From the following result in [9] it follows that it is necessary to introduce the concept of the geometric type of a Markov partition:

Theorem (Bonatti, Langevin) . Consider two orientation-preserving, structurally stable diffeomorphisms f and f ‘ defined on closed orientable surfaces M ² , and two saddle basic sets Л and Л ’ for f and f ‘ , respectively. If the basic sets Л and Л ’ admit Markov partitions of the same geometric type, then there exist canonical neighborhoods A(f, Л) and ^(f ‘ , Л ’ ), as well as a homeomorphism defined on these neighborhoods that conjugates f | △ ( f, л) and f ‘ | ^( f ‘ , л ‘ ) ■

The above theorem shows that the dynamics of a structurally stable 2-diffeomorphism in a neighborhood of a saddle basic set can be characterized by a finite combinatorial object, namely the geometric type of any Markov partition for Л.

In the situation where there is no prior information about the basic sets of the diffeomorphism, it is necessary to consider the concept of a Markov partition without using information about the basic set. For this purpose, we define an abstract Markov partition as introduced in [9]:

Definition 7. An abstract Markov partition is a pair ( R ,f ), where f is an orientationpreserving diffeomorphism of a surface M ² , and R = {R1 ,... ,R _m } is a finite sequence of pairwise disjoint rectangles embedded in the orientable surface M ² , such that these rectangles satisfy property (*).

In this paper we study basic sets whose Markov partition consists of a single rectangle. For simplicity of exposition, we will consider only these Markov partitions. An example of such a partition is the Markov partition for the Smale horseshoe [22] (see Fig. 4).

Fig. 4. The Markov partition of the Smale horseshoe with a single rectangle R .

The geometric type of an abstract Markov partition is defined according to the approach presented in [9]. The definition is also applicable to a Markov partition that is not abstract.

Consider an abstract Markov partition ( R ,f ), where R consists of a single rectangle R:

• •    choose an orientation R for one horizontal boundary of the rectangle R ;

• •    the orientation ш s determines an orientation ш ^и for one vertical boundary of the rectangle R (through the orientation of the surface into which this rectangle is embedded);

• •    the orientations ^ s and ш ^и allow us to speak of the bottom, top, left, and right boundaries of the rectangle R , as well as of any horizontal or vertical subrectangle.

Definition 8. Let V 1 ,..., V _h denote the connected components of f (R) A R in the order determined by the orientation ω ^s . By the definition of a Markov partition, these connected components are the images of horizontal subrectangles of the rectangle R , which we denote by H 1 , . . . , H _h in the order determined by the orientation ω ^u .

For each i ^ h there exists a unique natural number j such that f (H i ) A R = V j ; in this case we set Ф( i ) = j . Furthermore, we define E(i) = 1 if f maps the lower boundary of the subrectangle H i to the lower boundary of the subrectangle Vj ; otherwise we set s(i) = — 1. Thus, we define a one-to-one correspondence Ф : { i : i ^ h} —> {j : j <  h } , and a map e : {i : i <  h} —> { 1 , — 1 } .

The geometric type T of an abstract Markov partition ( R , f ) with a single rectangle R is defined by the integer h, the one-to-one correspondence Ф, and the map e . ²

For example, the Markov partitions considered in this paper have the following geometric type:

Definition 9. The geometric type of an abstract Markov partition ( R , f ) with a single rectangle R = {R} is called serpentine if:

• •    the map E(i) alternates sign;

• •    the map Ф( i ) = i .

Definition 10. A geometric type T of an abstract Markov partition is called realizable if there exists an orientation-preserving structurally stable diffeomorphism of a closed surface for which the saddle basic set admits a Markov partition with geometric type T .

In [9] a criterion is given that allows one to determine whether the geometric type of a Markov partition ( R , f ) is realizable. The formulation of this criterion is based on the construction of a surface with marked points, denoted by Xu ( R ,f ). Let ( R ,f ) consist of a single rectangle R , and let ∂ ^s R denote the union of the stable (horizontal) boundaries of this rectangle. This boundary is a closed curve with the previously chosen orientation ω ^s . A vertex v of the rectangle R is called stable–unstable if, when moving along the oriented boundary, it corresponds to a transition from a stable (horizontal) side of the rectangle R to an unstable (vertical) one. Otherwise, the vertex is called unstable–stable (see Fig. 5).

ds(R) Rnf-\d\R^

unstable-stable vertices stable-unstable vertices d^sR = [v₄;wi] и [v₂-,v₃]

Fig. 5. Construction of the surface with marked points X u (R , f ) .

Construction of the surface with marked points Su (R ,f):

Let R^u = cl( R \ f - 1 ( R )) be the closure of the set R \ f ^{- 1} (R); this is a union of finitely many strict horizontal subrectangles of the rectangle R . Then the set of stable boundaries d ^s R^u is the union of d ^s R and R A f - 1 (d ^s R) (see Fig. 5).

We identify each point x e R A f ^-1 ( d ^s R) with its image f (x) e f (R) A d ^s R. The quotient space thus obtained is a surface with boundary, which we denote by S ^u ( R , f ) (see Fig. 6). Let П ^и be the natural projection of R ^u onto S ^u ( R , f ).

Fig. 6. The surface with boundary S u ( R , f ) for the Smale horseshoe.

We call stable-unstable marked points of the surface S ^u ( R , f ) those that are images under n ^u of stable-unstable vertices of the rectangle R . Similarly, we define unstable-stable marked points of the surface S ^u ( R , f ).

Definition 11. A connected component of the boundary of the surface with marked points S ^u ( R ,f ) is called balanced if it contains an equal number of stable-unstable and unstable–stable marked points.

Definition 12. A connected component S of S ^u ( R ,f ) is called simple if it corresponds to one of the following two cases:

Case I. S has genus ³ equal to 1, and all connected components of the boundary of S are balanced.

Case II. S has genus equal to 0, and at most two connected components of the boundary of S are not balanced.

Definition 13. A surface with marked points S ^u ( R ,f ) is called simple if all of its connected components are simple.

For example, the surface with marked points S ^u ( R ,f ) constructed in Fig. 6 is simple: it is connected, has genus zero, and one of its three connected components of the boundary is balanced.

The following theorem from [9] provides a criterion for the realizability of the geometric type of an abstract Markov partition. We will use it in the proof of the realizability of the serpentine geometric type.

Theorem 2. The geometric type of an abstract Markov partition ( R ,f ) is realizable if and only if the surface with marked points S ^u ( R , f ) is simple.

Example. Non-realizable geometric type. Consider an abstract Markov partition (R0, f0) (with a single rectangle) represented in Fig. 7. The surface with boundary Su (R0, f0) has genus 0, but none of the three connected components of the boundary of Su(R0, f0) is balanced; therefore, the geometric type of the abstract Markov partition (R0, f0) is not realizable. Note that in this case the rectangle R0 contains a zero-dimensional invariant hyperbolic set whose dynamics is topologically conjugated to the dynamics of the Smale horseshoe, but this set is not saturated hyperbolic and therefore is not a basic set.

Fig. 7. Example of a non-realizable geometric type of a Markov partition ( R 0 , f ₀ ) and the surface with marked points S u ( R ₀ , f ₀ ).

3.    Realizability of the Serpentine Geometric Type

Let (R , f ) be a Markov partition of serpentine geometric type, where R = {R} (i.e., with a single rectangle) and f : M ² 4 M ² is an orientation-preserving structurally stable diffeomorphism. Consider the class of all such Markov partitions. Denote the vertices of the rectangle R by A , B , C , D . By the definition of the serpentine geometric type, s ( i ) must alternate in sign and Ф( i ) = i. Consequently, variability arises from the number h of vertical subrectangles V i, i = 1 ,... ,h , and from the difference in the first terms of the alternating sequence, namely e(i) = ( — 1)i or s ( i ) = ( — 1)i +1 for i >  1.

Consider h = 2 k and h = 2 k — 1, that is, an even and an odd number of connected components of f ( R ) П R , respectively (see Fig. 8, cases 1, 2 and 3, 4, respectively). For each V i , i = 1 ,..., h , there exists a horizontal subrectangle H i, and by the definition of the correspondence Ф( i ) of the serpentine geometric type, we have H i ^ V i . The variability in the first terms of the alternating sequence e(i) occurs when the lower boundary of H1 maps either to the lower boundary of V 1 (see Fig. 8, cases 1 and 3) or to the upper boundary of V 1 (see Fig. 8, cases 2 and 4).

It is not difficult to observe that in some of the considered cases the restrictions of the diffeomorphisms to the rectangle R are locally topologically conjugated in the sense that there exists a homeomorphism d : R U f m ( R ) ^ R U f j (R) such that f j • d | R = d • f m | r , where j, m are the numbers of the possible cases of the mappings according to Fig. 8.

1 case

h = 2k-l

Fig. 8. Possible cases of the serpentine geometric type.

3 case

h = 2k

For example, in cases 3 and 4 the homeomorphism d corresponds to a rotation by π . In the next section it will be shown that the remaining cases are pairwise non-conjugated.

However, the introduction of the notion of the serpentine geometric type and its visual representation does not guarantee its realizability in the sense of the existence of a structurally stable diffeomorphism whose saddle basic set admits a Markov partition of this geometric type. As mentioned earlier, to prove realizability it is necessary to construct the surface with marked points Tu(R ,f ) and verify its simplicity.

Fig. 9 shows the construction of the surfaces T u ^ R f ) for each of the considered cases.

Statement 2. The serpentine geometric type is realizable.

• <1 We present a proof for the second case of the Markov partition (R , f 2 ). In the remaining cases, the reasoning is analogous.

On the one hand, the Euler characteristic of the surface equals x(S ^u (R , f2^ =1 — h .

On the other hand, the Euler characteristic is given by x(S ^u (R , /2) = 2 — 2 g — b , where g is the genus of the surface and b is the number of connected components of its boundary. For T ^u (R , f 2 ) the boundary has b = h + 1 connected components. Hence,

X^ uCR ,f 2 ) ) =2 — 2 g — (h + 1) = 1 — h ^ g = 0 .

Thus, the genus of the surface T ^u (R , f 2 ) is 0, and at most two boundary components are unbalanced (in Fig. 9 the red boundary components are unbalanced and the green ones are balanced). It follows that the second case of the serpentine geometric type is realizable.

By analogous arguments, the first and third cases of the serpentine geometric type are also realizable. ⊲

Fig. 9. Construction of the surfaces T u ( R , f j ) for each j-th case of the serpentine geometric type.

Thus, the geometric type of any Markov partition in the class under consideration is realizable. Accordingly, we partition this class into three subclasses, denoted [ f i ], [ f 2 ], and [ f 3 ], corresponding to the three cases described above.

Statement 3. Let g be a structurally stable diffeomorphism with a basic set Л д that admits a Markov partition with a single rectangle R = {R} of such a geometric type that e ( i ) alternates in sign and Ф( i ) = h — i + 1 (that is, H i H V h , H2 H V h - i, ..., H h H V i ). Then there exists a diffeomorphism f with a basic set Л f from the class under consideration such that g is locally topologically conjugated to f on Л д and Л f , where Л f is a basic set of serpentine type with Ф( i ) = i .

• < Indeed, consider the class of Markov partitions (R, g) of geometric type with alternating e(i) and with the map Ф(i) = h — i + 1 (see Fig. 10). There exists j G {1,2,3} and a homeomorphism d: R U g(R) H R U fj(R) corresponding to a reflection with respect to the vertical axis such that fj • d = d • g, where j G {1, 2, 3} are the numbers of the possible cases of the mappings according to Fig. 8. ⊲

  

  Fig. 10. Local topological conjugacy of representatives of Markov partitions of geometric types with $(i) = i and $(i) = h — i +1.

  
  
  
  
  
  
  
  

  h = 2k — 1

  Ф) = (-!)*

  
  
  
4.    Boundary periodic points

In other words, including the cases with Ф( i ) = h — i + 1 does not enlarge the class of Markov partitions under consideration. Thus, in what follows we will work only with cases 1, 2, and 3, and treat them separately.

Let Л be a zero-dimensional nontrivial basic set of a structurally stable diffeomorphism f of a surface M ² . According to the results of V. Z. Grines in [23], such a set of a structurally stable diffeomorphism has so-called boundary points. For a point p of any nontrivial basic set Л on the surface, the stable and unstable manifolds W _p ^s and W _p ^u are one-dimensional. Then the point p divides each of the invariant manifolds into two separatrices, that is, the connected components of the set W ^ \ p , a G { s,u } .

Definition 14. A periodic point p G Л is called a boundary point if at least one of its separatrices does not intersect Л. If this separatrix is stable (i. e., I s ⁰ C Wp), then the point is called an s-boundary point. If this separatrix is unstable (i. e., l u ⁰ C Wp ), then the point is called a u-boundary point . Note that a point can be both an s -boundary and a u -boundary point; in this case it is called an ( s, u ) -boundary point .

In [23] V. Z. Grines proved that at least one of the two separatrices of a boundary point of the same stability always intersects Л. Moreover, the set of boundary points is nonempty and finite. There is another definition ⁴ of a boundary point, which will be given according to [24].

Fig. 11. (s, u)-boundary point p of zero-dimensional basic set Л, W lUOc (p) divide V (P) into V 1 and V 2 ; Wf_oc (p) divides V (P) into V 1 ‘ and V 2 ‘ .

Definition 15. Let Л be a nontrivial basic set of a structurally stable 2-diffeomorphism. A saddle periodic point p G Л is called an s-boundary point if there exists a neighborhood V (p) of the point p such that the segment W lOc (p) of its stable manifold divides it into two open sets V 1 and V 2 , satisfying the following conditions:

• •    V = V 1 U V 2 U (W^(p) n V );

• •    V 1 n Л = 0 and V ₂ n Л = 0.

Similarly, a point is called a u-boundary point if such a division exists for W i U c ( p ). A point that is concurrently an s -boundary and a u -boundary point is called an (s,u)-boundary point (see Fig. 11).

This property of the boundary point’s invariant manifolds helps to single out on the surface a region consisting of wandering points. Indeed, consider the first subclass of Markov partitions of serpentine geometric type [f1 ] (see Fig. 8) and the preimage of any horizontal subrectangle f — ¹ (H j ). It intersects each rectangle H i in exactly one component, that is, H i П f - ⁱ (H j ), V i <  h , V j <  h , contains exactly one horizontal subrectangle. Then the second preimage intersects H i П f ^-i (H j ) in only one horizontal subrectangle. Take j = 1, since the preimages of H 1 are precisely those that help to determine the location of a boundary point of the basic set admitting a Markov partition of subclass [ f i ]. From the construction of this Markov partition one can single out a sequence of nested horizontal subrectangles whose vertical sides are contracting:

P i = H i ,

P2 = Hi n f-(ph) = Pi n f - 1 (Pi ) ,

P3 = Hi n f-HHi ) n f ^-i ( H i n f ^-i ( H i )) = P2 n f ^{- i} (P2 ) ,

...

P k = P k - i n f ^{- i} (P k - i ) , k G N .

Due to the nested structure of this sequence, there exists a unique limit, which is the segment [a; b]. Clearly, under the action of fi, the segment [a; b] contracts, i.e., f^([a; b]) G [a; b] for all n G N. By analogous reasoning, one can single out a sequence of nested vertical subrectangles converging to a segment [c; d], which contracts under the action of inverse iterations of fi, i.e., f-n([c; d]) G [c; d] for all n G N. Then the point p( = [a; b] n [c; d] is a fixed point, and [a; b] C Ws1, [c; d] C Wpx by the definition of invariant manifolds (see Fig. 12).

Moreover, taking into account the orientation of the vertical boundary of the rectangle R , the open horizontal subrectangle abAD , bounded by [a; b] and the lower boundary of the rectangle R , contains no points from Л. Indeed, for any point z in this subrectangle, there exists a neighborhood U _z such that f-ⁿ(U _z ) П U _z = 0 for all n G N. Hence, abAD consists of wandering points.

Similarly, the open vertical subrectangle bounded by [ c ; d ] and the left boundary of the rectangle R contains no points from Л. This means that the point p1 is a boundary point by definition 15.

Then the manifold W ^s 1 is the union of the inverse iterations of the segment [a; b ]: W ^s 1 = UX =₀ f -ⁿ ([a; b]), and W u 1 is the union of the forward iterations of the segment [ c ; d ]: W u 1 = U” o f n ([c; d ]) .

In a similar way, one can find the second boundary point p 2 , locating at the intersection H h ∩ V h, and its stable and unstable manifolds W _p ^s ₂ and W _p ^u ₂ , respectively. From the construction of the mapping it follows that the points p 1 and p 2 are fixed points.

Fig. 12. Location of the boundary point p 1 .

Fig. 13 shows the forward and backward iterations of the initial rectangle UX   _x f i (R)

∀ n ∈ N.

Fig. 13. Construction of the Markov partition ( R , f 1 ) with the location of the boundary points p 1 , p 2 and their invariant manifolds.

Statement 4. The points p i and p2 are the only boundary points in the basic set Л admitting the Markov partition (R , f 1 ).

⊳ We prove that there are no boundary points other than p 1 and p 2 .

From the structure of the nested vertical and horizontal subrectangles it follows that Л is a subset of the closed rectangle bounded by the limit segments [ a ; b ] C W p 1 , [ c ; d ] C W pU 1 , and the analogous segments of the invariant manifolds of the point p 2 . Two of its vertices are p 1 and p 2 . Denote them as P 1 and P 2 respectively, so that the rectangle is noted as P1KP2M , moreover, PiKP2M C R . Consequently, there are no points of the basic set Л outside P 1 KP 2 M .

Points lying on Wp, , a E { s,u } , cannot be periodic (by the definition of Wp. ) and hence cannot be boundary.

Recall that the separatrices of the boundary points P 1 and P 2 bound the closed rectangle P1KP2M , which contains Л entirely. Inside it, every intersection of stable and unstable manifolds of distinct points of the basic set Л also belongs to Л. As shown in Fig. 14, for any periodic point q not lying on W p , a E { s,u } , all of its separatrices intersect the separatrices of the boundary points P1 and P ₂ (since every connected component of W ³ (Л) П R is an arc intersecting R horizontally, that is, connecting one vertical boundary to the other). Thus, the point q is not boundary by definition 14. ⊲

Fig. 14. A region that entirely contains Л, and a periodic point q that is not boundary.

In the previous section we stated that representatives of different subclasses of Markov partitions are not locally topologically conjugated. Below we prove this statement, relying on the differences in the structure of the sets of boundary periodic points.

Statement 5. Let f i and f j be diffeomorphisms from different subclasses [f i ] and [f j ], and let Л f _i , Л f j be zero-dimensional nontrivial basic sets corresponding to the Markov partitions (R , f i ) and ( R , f j ). Then there do not exist neighborhoods of Л f _i and Л f _j on which f i and f j are locally topologically conjugated.

⊳ Under local topological conjugacy a boundary point is mapped to a boundary point. Moreover, under the action of the conjugating diffeomorphism, the image of an orbit is an orbit. Let us examine which orbits constitute the set of boundary points for representatives of each subclass [ f i ], [ f 2 ], [ f a ].

• •    As mentioned earlier, in the first case, for the basic set of the Markov partition (R, fi), the set of boundary points consists of two fixed points.

• •    In the second case, for the basic set of the Markov partition (R, f2), there are also exactly two boundary points P₁^′and P₂^′, and the entire set of boundary periodic points consists of an orbit of period 2 (see Fig. 15).

• •    For the basic set of the representative of the third subclass [f3] of Markov partitions of serpentine geometric type (R, f3), there is only one fixed boundary point Pi’ (see Fig. 15).

5. Proof of the Nonexistence of an Energy Function

1 case

2 case

3 case

Fig. 15. Locations of the boundary points of the basic sets of the representatives of subclasses [f i ], f], and [f 3 ].

Thus, the diffeomorphisms from the subclasses [ f i ] , [ f 2 ] , [ f 3 ] are not locally topologically conjugated on their basic sets. ⊲

In this section we prove that if the basic set of a structurally stable diffeomorphism f : M ² ^ M ² admits a Markov partition of serpentine geometric type, and if R U f ( R ) lies in a disk, then f does not have an energy function. Note that it follows that f ^-1 ( R ) П R U f ( R ) also lies in a disk. We denote this disk by D .

In the previous section we demonstrated that the properties of invariant manifolds given in definitions 14 and 15 allow to identify a region that contains only wandering points. Since the sets of boundary points differ in the three cases of partitions, we consider each of them separately.

Let us now find the wandering points of the mapping f 1 . Recall that the previously considered rectangle P1KP2M C R contains the entire set Л, with P1M C W p 1 , P2M C W p 2 , KP 2 ⊂ W _P ^s ₂ , and KP 1 ⊂ W _P ^u ₁ .

In Fig. 16, the curves f ^{- i} (KP2) and f (P2M ) are shown, which bound the regions G and Q . These regions contain no non-wandering points. Due to the invariance of the manifolds W _P ^u ₁ and W _P ^u ₂ , W _P ^s ₁ and W _P ^s ₂ , this property is preserved for all iterations of the open disks G and Q . It is easy to see that these disks do not intersect their own iterations and therefore consist only of wandering points.

Each of the boundaries of the disks Q and G contains a pair of points from Л ( t i , q i and t 2 , q 2 in Fig. 16), which are pairs of conjugated points. Accordingly, at each iteration the pair of images of these points is also a pair of conjugated points. Hence, the basic set Л of serpentine type contains infinitely many pairs of conjugated points belonging to the invariant manifolds of the boundary points P 1 and P 2 .

Let S denote the rectangle given by the intersection of the disks Q and G . The vertices of S belong to the set Л, and the boundary dS has the form dS = [ x i , X 2 ] ^s U [ x 2 , Х з ] ^и U [ х з , X 4 ] ^s U [ x 4 ,x i ] ^u and contains exactly four points { x i ,X 2 ,X 3 ,X 4 } from Л (see Fig. 16).

Fig. 16. Wandering regions G, Q, S = G A Q, and the pairs of conjugated points t 1 , q 1 and t 2 , q 2 .

For the remaining cases of Markov partitions of serpentine geometric type, wandering regions G , Q, and their intersection S = Q A G with the same properties can be identified in a similar way (see Fig. 17). Therefore, in the subsequent proof we will refer to these regions without explicitly distinguishing between the different cases of serpentine geometric type Markov partitions.

1 case

Fig. 17. Regions G, Q, and S = G A Q corresponding to the representatives of subclasses [ f i ] , [ f a ], and f ].

The structure of the Markov partitions and the geometry of the open disks of wandering points, described in the previous section, make it possible to apply directly the proof from [13]. For completeness, we reproduce it here.

• ⊳    We prove that there does not exist a function defined on the disk D whose set of critical points coincides with Л.

Assume that there exists an energy function ^ ( x ) for the diffeomorphism f . Without loss of generality, we assume that ^ (Л) = 0. By the definition of an energy function ^ ( x ), its set of critical points coincides with the non-wandering set NW(f ). Hence, on the set cl S \ { x i , x ₂ ,x ₃ ,x 4 } the function ^ ( x ) has no critical points. From Proposition 1 of Section 2 it follows that

• •    ^ ( x ) >  0 for all x e ( x i ,x ₂ ) ^s U ( x ₃ ,x ₄ ) ^s ,

• •    ^ ( x ) <  0 for all x e ( x ₄ , x1 ) ^u U ( x ₂ , x ₃ ) ^u .

The set ^ - 1 (0) A S may have a complicated structure, but by Lemma 2 of [12] it contains an arc l whose closure separates either ( x 1 ,x ₂ ) ^s from ( x 3 ,x ₄ ) ^s or ( x ₂ , X 3 ) ^u from ( x ₄ ,x1 ) ^u (see Fig. 18).

Fig. 18. Possible locations of the separating arc l.

Consider the case when the arc l separates (x ₂ ,хз)^и and ( x ₄ , x 1 ) ^u , as shown in Fig. 19. In the other cases the reasoning will be the same. Denote by θ the subset of Q bounded by the arc l and the arc [ х з ,x 1 ] ^s . Since В C Q , we have В A NW(f) = 0. Note that cl В is a disk, i.e., a compact set, and therefore the function ^ ( x ) attains on it its minimum ^ _min ( x ) and maximum ^ _max ( x ).

Fig. 19. The disk θ and the values of the function ϕ on its different parts and on its boundary.

On the boundary of the disk дВ = [x1 ,x 3 ] ^s U l the function ^ ( x ) >  0, whereas for all x € ( x ₂ ,x 3 ) ^u the function ^ ( x ) <  0 (see Fig. 19). Thus, ( x ₂ ,x 3 ) ^u C В, which implies that the minimum ^ _min ( x ) is reached not on the boundary of В but in its interior.

However, all interior points of the disk θ are wandering, and there exists a critical point inside θ where ϕ _mi_n is reached. That contradicts the definition of an energy function, so the assumption of its existence was incorrect. Thus we proved that if there is a Smale horseshoe among the basic sets of a diffeomorphism, then such a diffeomorphism has no energy function. >