Novel Shannon Graph Entropy, Capacity: Spectral Graph Theory

by

Garimella Ramamurthy

Report No: IIIT/TR/2015/-1

Centre for Security, Theory and Algorithms

International Institute of Information Technology
Hyderabad - 500 032, INDIA
October 2015
 NOVEL SHANNON GRAPH ENTROPY, CAPACITY:
SPECTRAL GRAPH THEORY

G. Rama Murthy
International Institute of Information Technology
Hyderabad, India
rammurthy@iiit.ac.in

ABSTRACT
In this research paper, an interesting probability mass function is associated with the vertices of
graph. One among the various possible entropies ( Shannon, Renyii, Tsallis etc ) is associated
with such a probability mass function. Specifically Shannon entropy is utilized to define a
novel graph entropy. Characterization of minimum and maximum Shannon entropy graphs is
discussed. By associating a symmetric stochastic matrix with the graph, novel Shannon Capacity
of a graph is defined. Several interesting results in spectral graph theory of structured graphs
are reported. New results related to sparsest and densest cut computation are reported ( without
invoking Cheeger‟s inequality ).
This research paper is organized in the
I. INTRODUCTION:
following manner. In Section 2, relevant
Directed / undirected, weighted / unweighted research literature is reviewed. In Section 3, a
graphs naturally arise in various applications. novel graph entropy is defined by associating
Such graphs are associated with matrices such a graph with an interesting and natural
as weight matrix, incidence matrix, adjacency probability mass function. Its properties are
matrix, Laplacian etc. Such matrices implicitly studied. In Section 4, several interesting results
specify the number of vertices / edges, adjacency related to spectral graph theory of structured
information of vertices graphs are discussed. In Section 5, new results
( with edge connectivity ) and other related related to sparsest / densest cut computation
information ( such as edge weights ). In recent are discussed. In Section 5, some interesting
years, there is explosive interest in capturing applications of such graph entropy are briefly
networks arising in applications such as social specified. The research paper concludes in
networks, transportation networks, bio- Section 7.
informatics related networks ( e.g. gene
regulatory networks ) using suitable graphs. 2. REVIEW OF RESEARCH
Thus, NETWORK SCIENCE led to important LITERATURE:
problems such as community extraction,
frequent sub-graph mining etc. In research literature, there are already
In research efforts related to large scale efforts to associate a probability mass function
network science, various interesting scalar with a graph and compute one among various
valued measures ( performance related ) are possible entropies ( such as Renyii entropy,
defined and utilized for making interesting Tsallis entropy etc ) [9,Simonyi]. Such efforts
inferences on large scale graphs. Specifically, resulted in interesting insights into network
graphs are associated with a relevant science. The essential utility of those
probability mass function and related measures definitions of graph entropy relies on how
are computed. This research paper is an effort well the associated probability mass function
in that direction. captures the uncertainity related to the graph
structure ( relevant to the application of
interest ).
The author innovated the concept of
vertex degree probability mass function by
normalizing the associated vertex degree
distribution by twice the total number of
number of edges ( or equivalently sum of all
the vertex degrees ). Then he naturally
proposed one among various possible graph
entropies by computing the Shannon entropy
of such a Probability Mass Function ( PMF ).
It naturally follows that such a PMF can be
utilized to compute other entropies such as
Renyii Entropy (leading to other definitions of
graph entropy ). It is expected that the
selected probability mass function captures
interesting uncertainity associated with the
structure of graph. Efforts are underway to
explore interesting ramifications of graph
entropy proposed by the author. In the
following sections, we study various properties
of graph entropy proposed by the author.
3. SHANNON GRAPH ENTROPY, GRAPH
CAPACITY : PROPERTIES:
We begin the discussion with a definition
Definition: Vertex degree distribution of an
undirected graph G = ( V, E ) is the
distribution of edges of the graph at various
vertices i.e.
the number of edges of the graph that
are incident at vertex „i‟ for . term graph entropy means Shannon Graph
It is easy to see that
We conceive the innovative idea of
normalizing
and arriving at a probability mass
function. Formally, we have the following
definition
Definition: Vertex Degree Probability Mass
Function is a probability mass
function, where
With such a probability mass function, one of
the various possible entropies ( i.e. definitions
such as Shannon, Tsallis, Renyii etc ) can be
associated and utilized as the definition of
corresponding graph entropy. Specifically, we
have the following definition.
Definition: Shannon entropy of an undirected
graph with Vertex Degree Probability Mass
Function is given by
Remark 1: By virtue of the above definition,
the novel graph entropy is endowed with all
the properties of Shannon entropy.
Specifically, all the axioms satisfied by
Shannon entropy function are satisfied by
such graph entropy.
Remark 2: In literature, other graph entropies
are shown to satisfy properties such as the
SUB ADDITIVITY property. All such known
properties of Shannon entropy are satisfied by
the Shannon Graph entropy.
In the following discussion, the
Entropy.
The following interesting
Lemmas specify the maximum and minimum
entropy graphs ( i.e. graphs which will have
maximum and minimum Shannon graph
entropy ).
Lemma 1: An undirected graph has maximum
Shannon entropy if and only if the vertex
degree of all the vertices is equal.
Proof:
 From the basic properties of Shannon
entropy, we necessarily have that the entropy
will assume maximum possible value when
the associated probability mass function is
Thus, by the definition of vertex degree
probability mass function, we require that
This will happen if and only if
Thus, the maximum Shannon graph entropy
value associated with such a probability mass
function is given by
Q.E.D.
For instance, ring connected graph and clique
( fully connected graph ) are examples of
undirected graphs having the maximum
Shannon entropy. We now discuss, when
maximum entropy graphs exist on „N‟
vertices.
Now we consider the characterization
of minimum entropy graphs.
Lemma 2: An undirected graph has minimum
Shannon entropy if and only if it is a
Hub-and-Spoke graph i.e. the vertex degree
probability mass function is given by
upto relabeling of the vertices.
Proof: It is well known that the Shannon
entropy of a probability mass function is
minimum when it is degenerate i.e it is given
by
( 1, 0, 0,…0 ) upto permutation.
But, trivially such a vertex degree probability
mass function corresponds to a disconnected
graph. Thus, the next best minimum entropy
graph is attained, when one of the nodes has
degree (N -1 ) and all other nodes have degree
„one‟ i.e. Such a graph corresponds to a Hub-
and-Spoke graph. Thus, the vertex degree
probability mass function of such a Hub-and-
spoke graph is given by
Q.E.D.
We now compute the Shannon graph entropy ,
H(G) of a minimum entropy graph ( Hub-and-
spoke graph ):
H(G) =
= 1+(N–1)+ (N-1)
= N+ (N-1)
Novel Shannon Capacity of a Graph:
In the following discussion, we
associate an undirected, unweighted
graph with a stochastic matrix ( that
could be considered as the channel
matrix of a discrete memoryless
channel ).
Let A be
the adjacency matrix of such as graph
and let D be the associated diagonal
matrix whose diagonal elements are the
vertex degrees. It is immediate that
is a symmetric stochastic matrix. It
could be considered as the channel
matrix of a Discrete Memoryless
Channel.
 For such a Discrete Memoryless
Channel, we can consider the Vertex
Degree Probability Mass Function of the
graph ( defined earlier ) as the Input
probability mass function and compute
the Channel Capacity. We defined such
a quantity as a novel Shannon
Capacity of graph. It should be noted
that Lovasz defined and studied other
type of Shannon capacity of graph
[4, Lov].
From the above discussion, it should
be clear that the above considered
novel Shannon Capacity can also be
defined for weighted graphs ( with all
the weights being nonnegative ). In
this case, there are two possible
stochastic matrices: one based on the
adjacency matrix and the other based
on the non-negative weight matrix.
Also, utilizing the vertex degree
probability mass functions of two
graphs, distance between them could
be defined and studied in the following
manner.
Distance between Graphs: K-L
Divergence:
Consider two graphs
whose vertex degree probability mass function
are denoted by One possible
distance measure between them is given by
the associated Kullback-Leibler distance D ( P ||
Q ) between them ( also known as relative
entropy or cross entropy ). It is given by
D( P || Q ) = . invariant.
Since D( P || Q ) is not symmetric, we
utilize the Jeffrey divergence between
them as a related distance measure.
An interesting distance between graphs
can be computed by choosing one of
the graphs to be a maximum /
minimum entropy graph.
In the case of trees ( graphs without a
cycle ), one of the graphs can be chosen
to be a line / bus connected and the
K-L divergence , Jeffrey divergence can
be computed.
Remark:
Shannon entropy, capacity of
unweighted / weighted directed graphs are
defined based on the above approach,
Properties of such graph measures are studied
[Rama3].
4. SPECTRAL GRAPH THEORY: NOVEL
GRAPH ENTROPY:
Spectral graph theory
deals with the study of properties of a
graph in relationship to the characteristic
polynomial, eigenvalues and eigenvectors of
matrices associated with the graph, such as
its adjacency matrix or Laplacian matrix.
We prove new results related to spectral
graph theory of structured graphs.
An undirected graph has a
symmetric adjacency matrix, A and hence all
its eigenvalues are real. Furthermore, the
eigenvectors are orthonormal. We have the
following definition
Definition: An undirected graph‟s
SPECTRUM is the multiset of real
eigenvalues of its adjacency matrix, A
Fact: While the adjacency matrix depends on
the vertex labeling, its spectrum is a graph
We provide an interesting proof of the above
fact.
We need the following well known theorem.
Rayleigh’s Theorem: The local optima
of the quadratic form associated with a
 symmetric matrix A ( i.e. on entropy graph is d. The associated
the unit Euclidean hypersphere eigenvectors is the
( i.e. { X : occur at the “all ones” vector. In the case of
eigenvectors with the corresponding value clique, „d‟ assumes the maximum
of the quadratic form being the possible value of „N-1‟ if no self
eigenvalue. loops are allowed at each of the
vertices. If self loops are allowed at
Lemma 3: Eigenvalues of the adjacency matrix the vertices, the value of „d‟ ( for a
of an undirected graph, A are invariant under clique ) assumes the maximum
relabeling of the vertices: possible value of „N‟. Also, in the
case of Ring connected graph (
Proof: By Rayleigh‟s theorm, eigenvalues of which has maximum entropy ), the
A are the local optimum of the associated value of d is 2.
quadratic form evaluated on the unit
hypersphere. Thus, we need to reason that the Suppose we consider a maximum
quadratic form remains invariant under entropy graph ( in which self loops
relabeling of the vertices. We have that are not allowed at all the vertices ).
The trace of the associated adjacency
matrix is ZERO.
Thus, the sum of all the eigenvalues
other than the spectral radius is „-d‟.
= + Such an adjacency matris is
….. + indefinite.
,
It is easy to show that the adjacency
matrices of all maximum entropy
where, for instance, { are the graphs are RIGHT CIRCULANT
vertices connected to the vertex 1 ( one ) ( and matrices. From linear algebra
similarly other vertices ). research literature, there are well
Now, from the above known results on eigenvalues,
expression, it is clear that the quadratic form eigenvectors of right circulant
remains invariant under relabeling of the matrices. Those results are invoked
vertices. Specifically, relabeling just reorders the in understanding the spectrum of
expressions.Thus, the eigenvalues of A remain maximum entropy graphs.
invariant under relabeling of vertices Q. E..D
By Rayleigh‟s Theorem, largest
We now consider the spectrum of eigenvalue of the symmetric adjacency
maximum entropy graphs. matrix is the maximum value of the
quadratic form associated with A
MAXIMUM ENTROPY GRAPHS: evaluated on the unit hypersphere. (
could be labeled as “energy” ).
By Lemma 1, the sum of all the rows
of adjacency matrix is a constant, say Line / Bus Connected Graphs:
„d‟. Thus, using the well known
results on spectral radius of a non- NOTE: It is clear that a ring
negative adjacency matrix, the connected graph has maximum
maximum eigenvalue of a maximum
 Shannon graph entropy. A practical
example could be ring connected
molecules forming a chemical
compound. Suppose, we are interested Thus, we have the following:
in a graph with Shannon entropy
being as large as possible, but the Case I : N is odd. Recursively computing the
graph doesnot have a cycle in it. determinants, we have that
Then, the closest approximation is a
LINE connected graph. It is Case II : N is even . Recursively computing
immediate that the adjacency matrix the determinants, we have that
of a LINE/BUS connected graph on
N vertices is of the following form:

Thus, in this case the determinant value

oscillates between values +1 and -1 beginning
with -1.

Claim: Tree connected graph that has Hence, when N is even the adjacency matrix
maximum Shannon entropy is a line / bus A is unimodular
connected graph. Q.E.D.

Goal: To compute the determinant of We briefly consider the eigenvalues of line /

„N x N‟ adjacency matrix of a LINE / BUS bus connected graphs.
connected graph when the dimension N is
odd / even. For N=2, the eigenvalues of line
connected graph are +1, -1. Also ,
Lemma 4: The adjacency matrix of a line when N=3, the eigenvalues are 0,
connected graph, A is UNIMODULAR when .
the dimension, N is even ( i.e. Det (A) =
and is singular ( i.e. Det(A) = 0 ) when There is an interesting recursive
the dimension, N is odd. relationship between the characteristic
polynomial of line connected graphs.
Proof: Let denote the adjacency matrix Let CP( denote the characteristic
of a line connected graph on „j‟ vertices. polynomial of line connected graph on
„j‟ vertices. The following result holds
From the definition of determinant of
A, it is clear that when N is even as
well as odd

.
MINIMUM ENTROPY GRAPHS:
Also, as the initial condition, we have
Now, we consider minimum entropy
graphs and study the spectrum of such
graphs. From Lemma 2, we know that it
must be a Hub-and-Spoke graph. The
adjacency matrix A of such a graph is of
the following form:
A= = ,
Where is an ( N-1 ) dimensional
column vector of all ones. The following
results can easily be verified
When N =2, the eigenvalues of A are
+1,-1
Trace ( A ) = 0 . Thus A is indefinite
matrix
Determinant (A) = 0 when N > 2.
Now, we have the following goal.
Goal: To determine all the eigenvalues,
eigenvectors of the adjacency matrix A
associated with a Hub-and-spoke graph
when N ( the number of vertices ) is
strictly greater than 2.
Lemma 5: The eigenvalues of the graph
associated with a minimum entropy graph
on N vertices are
the
multiplicity of the eigenvalue at 0 is (N-
2).
Equivalently, the characteristic polynomial
of A is given by
Proof: We prove the result by
mathematical induction. Consider the
case where N = 3.
Thus, the eigenvalues are 0, ,- .
Hence the result holds true for the case
N =3.
Now, the induction hypothesis for
the case where the dimension of A is (N-1)
specifies that the characteristic polynomial of
A is given by
.
We now show that the above induction
hypothesis, gives the result for the case where
the dimension of A is N.
Using the induction hypothesis and the
structure of the matrix on the RHS in the
above expression, we have that
-
.
The above result holds for the case when N
is even as well as odd. Thus, the
characteristic polynomial of A is given by
.
Hence, the eigenvalues of A are given by
the
multiplicity of the eigenvalue at 0 is (N-2).
Q.E.D.
Characterization of Null Space of A :
Let N be odd. Consider the vectors in
which the components out of
(N-1) components are +1 and the other
components are -1. Also, let the
last component be ZERO. All such
vectors are in the null space of
adjacency matrix of minimum Shannon
entropy graph when N is odd.
Now let N be even. Any one
component among the first (N-1)
components is zero.
Also, consider the vectors in which the
components out of (N-2)
components are +1 and the other
components are -1. Further, let the last
component be ZERO. All such vectors
are in the null space of adjacency
matrix of minimum Shannon entropy
graph when N is even
Eigenvectors of A corresponding to
Non-zero Eigenvalues:
By the definition of
eigenvector, it can be shown that the
column vector whose first (N-1)
components are +1 and the last
component is is the
eigenvector corresponding to eigenvalue
. In the case of eigenvalue
the last component of
eigenvector is . on
Eigenvectors of A: Corners of Unit
Hypercube:
Suppose a column vector
is in the null space of A. Then we
have that
Thus,
Depending on whether Sign ( 0 ) = +1
or -1,
the only vector in the vector in the
null space of A that can also be stable
state is the “all-ones vector” or „all-
minu-ones vector”.
Consider a corner of hypercube, with
the last component being +1. Then, we
have that the first ( N-1) components
of A are all equal to „one‟. Thus,
cannnot be an eigenvector since, the
last component of A will be equal to
( N -1 ). Similarly, when the last
component of is -1, we reason that
cannot be an eigenvector.
Claim: None of the corners of
hypercube can be an eigenvector of A.
5. SPARSEST CUT COMPUTATION in
DIRECTED / UNDIRECTED
UNWEIGHTED GRAPHS : SPECTRAL
GRAPH THEORY:
Traditionally, Cheeger’s inequality is
utilized to approximate the sparsest
cut of a graph through the second
eigenvalue of its Laplacian. It is
considered to be the most important
Theorem in spectral graph Theory and
one of the most useful facts in
algorithmic applications.
We now
provide a different approach to the
problem by invoking the known results
minimum cut computation in
directed / undirected graphs associated
with a Hopfield neural network. In the
following we provide a review of the
relevant literature for the convenience
of the reader [Rama1].
REVIEW OF RESEARCH LITERATURE: MIN /
MAX CUT COMPUTATION : HOPFIELD
NEURAL NETWORK
Contribution of Hopfield et.al:

Hopfield neural network constitutes a which state updation as in (5.1) is
discrete time nonlinear dynamical system. It
is naturally associated with a weighted
undirected graph G = (V,E), where V is the
set of vertices and E is the set of edges. A
weight value is attached to each edge and a
threshold value is attached to each vertex/node
/artificial neuron of the graph. The order of
the network is the number of nodes / vertices
in the associated graph. Thus a discrete time
Hopfield neural network of order „N‟ is
uniquely specified by [3, Hop]
(A) N x N Symmetric Synaptic Weight
Matrix M i.e. denotes the weight
attached to the edge from node i to node
j ( node j to node i )
(B) Nx1 Threshold Vector i.e. denotes the
threshold attached to node „i‟.
Each neuron is in one of the two states i.e.
+1 or -1. Thus, the state space of such a non-
linear dynamical system is the N-dimensional
unit hypercube. For notational purposes, let
denote the state of node / neuron „i‟ at
the discrete time index „t‟. Let the state of the
Hopfield neural network at time „t‟ be
denoted by the Nx1 vector The state
at node „i‟ is updated in the following
manner ( i.e. computation of next state of
node „i‟ )
…..(5.1)
i.e. the next state at node „i‟ is +1 if the
term in the bracket is non-negative and -1
if it is negative. Depending on the set of
nodes at which the state updation given in
equation (5.1) is performed, the neural
network operation is classified into the
following modes:
Serial Mode: The set of nodes at
performed is exactly one i.e. at time
„t‟ only at one of the nodes / neurons
the above state updation is performed.
Fully Parallel Mode: At time „t‟ , the
state updation as in (5.1) is performed
simultaneously at all the nodes
In the state space of discrete time Hopfiled
neural network, there are certain distinguished
states, called the STABLE STATES.
Definition: A state V(t) is called a “Stable
State” if and only if
….(5.2)
Thus, if state dynamics of the network
reaches the stable state at some time „t‟, it
will remain there for ever i.e. no change in
the state of network occurs regardless of the
mode of operation of the network ( i.e. it is a
fixed point in the state dynamics of discrete
time Hopfield neural network ).
The following Convergence
Theorem summarizes the dynamics of
discrete time Hopfield neural network in the
serial and parallel modes of operation. It
characterizes the operation of the neural
network as an associative memory.
Theorem 1: Let the pair Z = ( M,T )
specify a Hopfield neural network. Then the
following hold true:
[1] Hopfield : If N is operating in a serial
mode and the elements of the diagonal of M
are non-negative, the network will always
converge to a stable state
( i.e. there are no cycles in the state space
).
[2] Goles: If N is operating in the fully
parallel mode, the network will always
converge to a stable state or to a cycle of
length 2 ( i.e the cycles in the state space are
of length ).
The proof of above
Theorem is based on associating the state
dynamics of Hopfield Neural Network (HNN)
with an energy function. It is reasoned that
the energy function is non-decreasing when
the state of the network is updated ( at
successive time instants ). Since the energy
function is bounded from above, the energy
will converge to some value. The next step
in the proof is to show that constant energy
implies a stable state is reached in the first
case and atmost a cycle of length 2 is
reached in the second case.
The so called
energy function utilized to prove the above
convergence Theorem is the following one:
.(5.3)
Thus, HNN, when operating in the serial
mode will always get to a stable state that
corresponds to a local maximum of the
energy function. Hence the Theorem suggests
that Hopfield Associative Memory (HAM)
could be utilized as a device for performing
local/global search to compute the maximum
value of the energy function.
Contribution of Bruck et.al
The
above Theorem implies that all the
optimization problems which involve
optimization of a quadratic form over the unit
hypercube ( constraint / feasible set ) can be
mapped to a HNN which performs a search
for its optimum. One such problem is the
computation of a minimum cut in an
undirected graph.
For the sake of
completeness, we now provide the definition
of a “cut” in a graph
Definition: Consider a weighted, undirected
graph, G with the set of edges E and the set
of vertices V i.e. G = ( V, E ). Consider a
subset U of V and let The set of
edges each of which is incident at a node
in U and at a node in is called a cut
in G.
Definition: The sum of edge weights of a
cut is the weight of the cut. From among all
possible cuts, the cut with minimum weight
is called a Minimum cut of the graph.
Note: Thus, min cut specification is equivalent
to specifying the subset of vertices, on one
side of the cut.
Minimum and Maximum Cut in
Undirected Graphs: Stable / Anti-
Stable States:
In the following Theorem,
proved in [1, BrB], the equivalence between
the minimum cut and the computation of
global optimum of energy function of HNN
is summarized.
Theorem 2: Consider a Hopfield Neural
Network (HNN)
Z= (M,T) with the thresholds at all nodes
being zero i.e. T 0. The problem of finding
the global optimum stable state ( for which the
energy is maximum ) is equivalent to finding
a minimum cut in the graph corresponding to
Z.
Corollary:
Consider a Hopfield neural network
Z =(M,T) with the thresholds at all neurons
being zero i.e. T . The problem of finding
a state V for which the energy is global
minimum is equal to finding a maximum cut
in the graph corresponding to Z.
Proof: Follows from the argument as in
[1, BrB]. We repeat the argument for clarity (
required for understanding the algorithms for
MIN/MAX computation in DIRECTED /
UNDIRECTED graphs ).
It is proved in [8, RaN], there is no
loss of generality in assuming that the
threshold at all the nodes/vertices of the
Hopfield neural network is zero. Thus, the
energy function is a pure quadratic form i.e
. Also, the local minimum values attained at
Let us denote the sum of weights of edges
in the graph with both end points in vertex
set as and let , denote
the corresponding sums for the other two
cases. We readily have that
( stable state, S of the associated Hopfield
. neural network N = (M,T) with T
Since the first term in the above equation is
constant ( it is the sum of weights of all
edges ), it follows that the minimization of H
is equivalent to maximization of . It is
clear that is the weight of the cut in N
with being the nodes of the Z with state
equal to 1 . Q. E.D.
Thus, the operation of Hopfield neural
network in the serial mode is equivalent to
conducting a local search algorithm for
finding a minimum cut in the associated
graph. State updation at a node in the
network is equivalent to moving it from one
side of the cut to the other side in the local
search algorithm.
As in the case of stable states, we have the
following definition.
Definition: The local minimum vectors of a
quadratic form on the hypercube are called
anti-stable states. This concept w as first
introduced in [8, RaN].
We have the following definition associated
with the values of quadratic form ( energy
function ) at the stable states and anti-stable
states.
Definition:
Consider a quadratic form
optimized on the unit hypercube ( constraint
set ). The local maximum values attained at
the stable states are called the stable values.
the anti-stable states are called the anti-stable
values.
Remark 1 :
In view of Theorem 2 , minimum
cut computation in an undirected graph
reduces to determining the global optimum
. The { +
1 } components of S determine the subset of
vertices that are members of one side of the
cut ( and { - 1 } components determine the
vertices on the other side of the cut).
Similarly, maximum cut computation requires
determination of global optimum anti-stable state.
Minimum and Maximum Cut in
Directed Graphs: Stable / Anti-Stable
States:
In
view of the above Theorem, a natural
question that arises is to see if a Hopfield
neural network can be designed to perform a
local search for minimum cut in a directed
graph. In that effort we now provide formal
definition of minimum cut in a directed
graph.
Definition: Let a weighted, directed graph be
denoted by G= (V,E). Every edge has a
weight and direction associated with it. The
weights of directed edges can be represented
by an N x N matrix M in which
represents the weight of the edge from vertex
„i‟ to vertex „j‟. Let a subset of the vertex
set V be denoted by and and let
The set of edges each of
which has its tail at a vertex in and its
head at a vertex in is called a
DIRECTED CUT OF G. The directed cut
with minimum total weight is called a
DIRECTED MINIMUM CUT (DMC) in G.
The following Theorem is in
the same spirit of Theorem for directed
graphs.
Theorem 3: Let M be the matrix of edge
weights ( M is not necessarily symmetric ) in
a weighted directed graph denoted by G =
(V,E). The network performs a
local search for a DMC of G where
Proof: Refer [2,BrS].
Thus, this Theorem shows that the
computation of minimum cut in a directed
graph is equivalent to determining the global
optimum stable state of a suitably chosen
Hopfield neural network.
Note: Similarly as in Theorem 2, the
computation of directed maximum cut ( i.e.
maximum cut in a directed graph ) reduces to
computation of global minimum
anti-stable state ( global minimum vector of
quadratic energy function ) in a suitably
chosen Hopfield neural network.
Claim 1 : In summary MIN / MAX cut
computation
in directed / undirected graphs is equivalent
to the determination of global optimum
stable / anti-stable state computation of
associated Hopfield neural network.
Sparsest Cut Computation in
Undirected , Unweighted Graphs:
In view of Theorem 2, we have the
following claim.
Claim : Since the weight matrix of an
undirected graph is a { 0, 1 } matrix,
computation of local / global minimum cut
computation in the associated graph is
equivalent to computation of local / global
optimum stable state in the associated
Hopfield neural network. Specifically, the
global minimum cut computation corresponds
to determining the cut with smallest number
of edges i.e. the sparsest cut
Since, the adjacency matrix of the
undirected graph is a non-negative ( {
0, 1} ) matrix, the global optimum
stable state is the all-ones vector, It
corresponds to the “empty cut”.
Thus, computation of sparsest cut in
such a graph is equivalent to
determination of second largest stable
state.
From the proof of Theorem 2, the following
equation readily follows in the case of an
undirected, unweighted graph. ( with
associated { 0, 1} weight matrix )
Second Largest Stable Value ( SLSV ) =
2 ( Total number of edges ) - 4 ( weight of
Sparsest Cut ).
Equivalently, we have that
Number of Cut Edges in Sparsest Cut =
( 2 |E| - SLSV ).
Thus, ideally we would like to compute the
Second Largest Stable State and the Second
Largest Stable Value (SLSV). We specifically
consider the following case where such a
thing is possible. The proof of above Lemma is based on
projecting points, X on the unit hypercube
CASE I : The Vertex Degree of all vertices is onto the unit hypersphere, Y using the
a constant., say „d‟ i.e. “all ones vector ” is following transformation:
an eigenvector of adjacency matrix A
corresponding to the spectral radius, “d”.

In such a case, the “all ones-column is utilized.

vector”, denoted by .
is the eigenvector Hence, we have that
corresponding to the largest eigenvalue .
of „d‟. It also, happens to be the
Using the Rayleigh‟s Theorem, we have the
largest stable state and corresponds to
following inequality satisfied by the
empty cut ( by Theorem 2 ).
eigenvalues of W
In this case, the second largest stable
.
state corresponds to every vector on
the hypercube with exactly one
Hence, we have that
element being „-1‟. There are N such
.
vectors and all of them. The
corresponding graph corresponds to a
Lemma 7: Let W be a symmetric, non-
“Hub-and-Spoke graph” with any one
negative matrix and be the
of the vertices as the hub. Such a
all-ones column vector. The following
graph leads to sparsest cut with “d” cut
statements are true:
edges

CASE II: The Vertex Degree of all vertices
is NOT a constant i.e. “all ones vector ” is
NOT an eigenvector of adjacency matrix A.
In this case, the following Lemmas proved
in [Rama ] provides an approach to bound
the Second Largest Stable Value (SLSV).
is the eigenvector corresponding to
the spectral radius of W
or
Corner of hypercube can be the
eigenvector of W corresponding to the
smallest eigenvalue only, if is not an
eigenvector

Lemma 6: Let W be a symmetric non- Proof: Refer [6, Rama2].

negative matrix and be the spectral
radius of W. Then we have the following Let
lower bound on i.e.

and let

= 2 ( Total Number of Edges )

Proof: Refer [Rama ].
Case ( ii ) is not an eigenvector. Thus, by
Rayleigh‟s theorem all the stable values must
lie between the second smallest eigenvalue
and the smallest eigenvalue. Hence, we have
the following bounds on eigenvalues of A:
Hence, we have the following lower, upper
bounds on the Second Largest Stable Value
(SLSV). By Rayleigh‟s Theorem , we have
that
Using the earlier result on relating the SLSV
with the number of edges in sparsest cut
denoted by R, we have that
. with the vertex degree distribution of an
Densest Cut Computation in
Undirected , Unweighted Graphs:
Using the
Corollary of Theorem 2 on computation of
maximum cut, we have the following result:
Global Minimum Anti -Stable Value
GMASV ) =
2 (Total number of edges ) - 4 ( weight of
Densest Cut ).
Equivalently, we have that
Number of Cut Edges in Densest Cut =
( 2 |E| - GMASV ).
Remark: Using Theorem 3, similar results
can easily derived for computation of sparsest
and densest cut in directed graphs. Details are
avoided for brevity.
6. APPLICATIONS of SHANNON GRAPH
ENTROPY:
In network science ( applied to
social network analysis, transportation
networks etc ), untruncated / truncated,
normalized ( neglecting the vertices whose
degree is less than some threshold ) vertex
degree distribution is utilized to arrive at
various scalar measures. Shannon Graph
entropy of associated vertex degree probability
mass function can be utilized to determine
how “uniform‟/ “non-uniform” is the vertex
degree probability mass function. Also, various
moments associated with the vertex degree
PMF can be computed. Other information
theoretic measures can be associated with the
vertex degree PMF of “pruned graph”
7. CONCLUSIONS :
In this research paper, a
novel probability mass function is associated
undirected graph. Shannon entropy of a graph
is defined to be the entropy associated with
such a probability mass function.
Characterization of undirected graphs with
minimum and maximum Shannon entropy is
discussed. Results in spectral graph theory of
structured graphs are discussed. New results
( on sparsest / densest cut computation in
undirected / directed graphs are discussed.
REFERENCES:
[1] [BrB] J. Bruck and M. Blaum,”Neural
Networks, Error Correcting Codes and
Polynomials over the Binary Cube,” IEEE
Transactions on Information Theory, Vol.35,
No.5, September 1989.
 [2] [BrS] J. Bruck and J. Sanz, “ A Study on
Neural Networks, “ International Journal of
Intelligemt System, Vol. 3, pp. 59-75, 1988.

[3] [Hop] J.J. Hopfield, “Neural Networks and

Physical Systems with Emergent Collective
Computational Abilities,” Proceedings of National
Academy Of Sciences, USA Vol. 79, pp. 2554-2558,
1982

[4, Lov] L.Lovasz,”On the Shannon Capacity of

a Graph, “ IEEE Transactions on Information
Theory, IT, Vol. IT-25, January 1979

[5, Rama1] G. Rama Murthy,”Multi-dimensional

Neural Networks : Unified Theory, “ Research
monograph published by NEW AGE
INTERNATIONAL PUBLISHERS, NEW DELHI,
2007.

[6, Rama2] G. Rama Murthy,”Bounding the

Eigenvalues of a Matrix: Optimization of Quadratic
Forms, “ IIIT Technical report 2015

[7, Rama3] G. Rama Murthy, “Novel Shannon

Entropy, Capacity of a Graph: Applications,” IIIT
Technical Report in Preparation

[8, RaN] G. Rama Murthy and B. Nischal,” Hopfield-

Amari Neural Network : Minimization of Quadratic
forms,” The 6th International Conference on Soft
Computing and Intelligent Systems, Kobe Convention
Center (Kobe Portopia Hotel) November 20-24, 2012,
Kobe, Japan.

[9, Sim] G. Simonyi,”Graph Entropy: A

Survey,” DIMACS Series in Discrete
Mathematics and Theoretical Computer
Science