symmetries in entropy space - ece.drexel.eduece.drexel.edu/walsh/walsh_itw_2016.pdf ·...
TRANSCRIPT
Symmetries in Entropy Space
Jayant Apte, Qi Chen, John MacLaren Walsh
Department of Electrical and Computer Engineering
Drexel University
Philadelphia, PA
Thanks to NSF CCF-1421828
1
Co-Authors & Software Advertisement
Jayant Apte (Ph.D)Defended 10-Aug-2016
Dissertation presents symmetry exploiting algorithms for proving network coding capacity regions.
Try our software: ITAP & ITCP
Qi Chen, Ph.D. Ph.D. CUHK w/ R. W. Yeung
Postdoc @ Drexel Fall, 2015
Postdoc @ INC, CUHK, 2016
On the Job Market!
NEW IT&NC SOFTWARE TOOL RELEASES!!!! (ASPITRG homepage & GitHub)Information Theoretic Achievability Prover (ITAP):For a given network, determine entire rate region associated w/ a class of linear codes!Also, can determine achievability of a specified rate point, bounds for linear secret sharing,and test where a specified polymatroid is linear over a specified field.Information Theoretic Converse Prover (ITCP):For a given network coding problem, computes polyhedral cones outer bounding its capacityregion using a custom new symmetry exploiting polyhedral projection algorithm!Both Shannon & non-Shannon outer bounds are supported. Also works with secret sharing.
http://www.ece.drexel.edu/walsh/aspitrg/software.html
Outline
1. Entropy Region ¯
�
⇤N
& Shannon Outer Bound �N
2. Portion Symmetric Under Group G, ¯
⇤G
and G
3. Power-Set Orbit Structures & Hierarchical Results
4. ¯
⇤G
?=
G
for N = 4, 5 (exhaustive) and N � 6.
Entropy Region ¯
�
⇤N
& Shannon Outer Bound �N
1. X = (X1, . . . , XN
) N discrete RVs
2. every subset XA = (Xi
, i 2 A) A ✓{1, . . . , N} ⌘ [N ] has joint entropy
h(XA).
3. h = (h(XA) |A ✓ [N ] ) 2 R2N �1en-
tropic vector
• Example: for N = 3, h =
(h1, h2, h3, h12, h13, h23, h123).
4. a ho
2 R2N �1 is entropic if 9 joint PMF
pX s.t. h(pX) = ho
.
5. Region of entropic vectors = �
⇤N
6. Closure ¯
�
⇤N
is a convex cone
h2
h1
h12
Shannon bound �N
: h 2 R2N �1 s.t.
hA + hB � hA[B + hA\B (1)
hA[B � hA (2)
i.e. submodular & non.-dec.
for N 2 {2, 3}, �n
=
¯
�
⇤N
,
but for N � 4, ¯
�
⇤N
( �N
.
¯
�
⇤N
is an unknown non-polyhedral convex cone for N � 4.
determining capacity regions of all networks under network coding , complete
characterization of ¯
�
⇤N
Entropy Region ¯
�
⇤N
– Why Care?
• All fundamental laws of information theory
• Network Coding Capacity Region (Streaming, Distributed Storage, & Caching)
• Implications among conditional independences (Graphical Models & Machine
Learning)
• Fundamental Limits for Secret Sharing Schemes
• More (inequalities for subgroup sizes, matrix rank inequalities, Kolmogorov
Complexities, etc.)
Group Action & Symmetries
h2
h1
h12
(1,0,1)
(0,1,1)
⇡ = (12)
Any permutation ⇡ 2 Sn
, ⇡ : {1, . . . , n} ! {1, . . . , n} a bijection, is a symmetry of ¯
�
⇤n
.
Under ⇡, the RVs (X1, . . . , Xn
) 7! (X⇡(1), . . . , X⇡(n)),
so hA 7! h⇡(A), with ⇡(A) := {⇡(i)|i 2 A}.
Example: ⇡ = (12),
⇡([h1, h2, h12, h3, h13, h23, h123]T
) = [h2, h1, h12, h3, h23, h13, h123]T .
Clearly, if h 2 ¯
�
⇤n
or �n
, ⇡(h).
(definition is insensitive to ordering of RVs)
Key Question: ¯
⇤G
?=
G
S2 =
⇤S2
Fix
S2
h1
h2
h12
What about those h 2 ¯
�
⇤n
fixed under a group of problem symmetries G ✓ Sn
?
Fix
G
(H) := {h 2 H|⇡(h) = h 8⇡ 2 G} (3)
Define G
= Fix
G
(�
n
) and ⇤G
= Fix
G
(�
⇤n
).
For what types of problem symmetries can we expect Shannon-type inequalities to be
su�cient? i.e., for which G does ¯
⇤G
=
G
?
Orbits in the Power Set
N = {1, . . . , n}
Fix
G
, and hence ¯
⇤G
and G
on depend on G through 2
N //G – the orbits in the
power set.
• ⇡(h) = h 8⇡ 2 G () hA = h⇡(A)8⇡ 2 G, 8A ✓ N .
• OG
(A) := {⇡(A)|⇡ 2 G}. hA = hB 8B 2 OG
(A)
• OG
=
�O
G
(A)|A 2 2
N , the power set orbits. (Partitions 2
N )
Note: Multiple groups can yield the same power set orbits.
i.e. can have OG
= OG
0 for G 6= G0.
Power Set Orbits, N = 4
{;}
O(1)
O(12)
O(123)
{N}
(a) OS4 = OA4
{;}
O(1)
O(12)
O(123)
{N}
O(13)
(b) OD4 = OC4
{;}
O(1)
O(123)
{N}
O(13)
O(14) O(12)
(c) OV =h(12)(34),(13)(24)i
{;}
O(1)
O(123)
{N}
O(3)
O(134)
O(13)O(14)
O(12)O(34)
(d) OS24=h(12)(34)i
Figure 1: Indecomposable power set orbits on N = {1, 2, 3, 4}
Power Set Orbits under di↵erent groups are also ordered by refinement of associated
partition of the power set.
Power Set Orbits, N = 5
{;}
O(1)
O(12)
O(123)
O(1234)
{N}
(a) OS5 = OA5 = OGA(1,5)
{;}
O(1)
O(12)
O(123)
O(1234)
{N}
O(13)
O(124)
(b) OD5 = OC5
Figure 2: Indecomposable orbit structures on N = {1, 2, 3, 4, 5}
Observe that while a poset, need not be a lattice.
¯
⇤G
?=
G
– Some Implications
• If G G0, then OG
OG
0 , i.e. partition OG
refines OG
0
– (extra group elements can force more equivalences)
Thm. 2: If OG
OG
0 , and ⇤G
=
G
, then ⇤G
0 =
G
0 .
• Equiv., if OG
OG
0 , and ⇤G
0 ( G
0 , then ⇤G
( G
.
Chen & Yeung – Partition Symmetrical Entropy Functions:¯
⇤G
?=
G
for G = Sn1 ⇥ S
n2 ⇥ · · · ⇥ Snk
Thm. 1 (Qi Chen & R. W. Yeung): Let p = {N1, N2, . . . , Nt
} be a t-partition of N ,
and Gp
= SN1 ⇥ SN2 ⇥ · · · ⇥ SNt . For |N | � 4,
⇤Gp
=
Gp if and only if p = {N} or
{{i}, N \ {i}} for some i 2 N .
What about general ¯
⇤G
?=
G
? Remaining cases?
• For any G, let p = N//G = {{⇡(i)|⇡ 2 G}|i 2 N}, then G Gp
– Thm. 1: ¯
⇤G
( G
unless N//G = {N} or N//G = {i, N \ {i}}, i 2 N .
• N//G = {N}: G is transitive.
• N//G = {i, N\}: G = S1 ⇥ G0,
– G fixes some i, then G0 on remainder is transitive on N \ {i}.
• Priority (Pessimistic) – Maximal Transitive Subgroups
• Priority (Optimistic) – Minimal Transitive Subgroups
¯
⇤G
?=
G
– Another Implication
Thm. 3: Let G ⇥ S1 act on N [ {n + 1} (fix n + 1, G acts on N ). Then
⇤S1⇥G
=
S1⇥G
=)
⇤G
=
G
.
Proof:
• orbits of OG⇥S1 = each orbit in O
G
repeated twice: once w/o n + 1, and once w/
n + 1.
• projN G⇥S1 =
G
– ✓: First 2
N � 1 coordinates are same subsets obeying same inequalities, must
also have the remaining coordinates exist obeying more inequalities.
– ◆: Take h 2 G
, extend to h0 on N [ {n + 1} via h0A = hA\N , then
h0 2 G⇥S1 .
• projN ⇤G⇥S1
=
⇤G
– ✓: Take X1, . . . , Xn
from X1, . . . , Xn+1 realizing h0 2 ⇤G⇥S1
. Realizes a
h 2 ⇤G
.
– ◆: Extend as previous, Xn+1 = 0, deterministic.
¯
⇤G
?=
G
– Complete Answer for N = 4
Symmetric & Alternating:
S4 : �(1234), (12)�A4 : �(123), (12)(34)�
Dihedral & Cyclic:
D4 : �(1234), (13)�C4 : �(1234)�
Normal Klein 4-group:
V : �(12)(34), (13)(24)�
Double Transp.:
S24 : �(12)(34)�
(1, 3)-Partition:
S1 ⇥ S3 : �(234), (23)�S1 ⇥ A3 : �(234), (243)�
(2, 2)-Partition:
S2 ⇥ S2 : �(12), (34)�
(1, 1, 2)-Partition:
S1 ⇥ S1 ⇥ S2 : �(34)�
Trivial: �()�
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
=
⇤G
G
=
⇤G
G
=
⇤G
G
=
⇤G
Power Set Orbits, N = 4
{;}
O(1)
O(12)
O(123)
{N}
(a) OS4 = OA4
{;}
O(1)
O(12)
O(123)
{N}
O(13)
(b) OD4 = OC4
{;}
O(1)
O(123)
{N}
O(13)
O(14) O(12)
(c) OV =h(12)(34),(13)(24)i
{;}
O(1)
O(123)
{N}
O(3)
O(134)
O(13)O(14)
O(12)O(34)
(d) OS24=h(12)(34)i
Figure 1: Indecomposable power set orbits on N = {1, 2, 3, 4}
Power Set Orbits under di↵erent groups are also ordered by refinement of associated
partition of the power set.
Power Set Orbits, N = 4
{;}
O(1)
O(12)
O(123)
{N}
(a) OS4 = OA4
{;}
O(1)
O(12)
O(123)
{N}
O(13)
(b) OD4 = OC4
{;}
O(1)
O(123)
{N}
O(13)
O(14) O(12)
(c) OV =h(12)(34),(13)(24)i
{;}
O(1)
O(123)
{N}
O(3)
O(134)
O(13)O(14)
O(12)O(34)
(d) OS24=h(12)(34)i
Figure 1: Indecomposable power set orbits on N = {1, 2, 3, 4}
Power Set Orbits under di↵erent groups are also ordered by refinement of associated
partition of the power set.
Power Set Orbits, N = 4
{;}
O(1)
O(12)
O(123)
{N}
(a) OS4 = OA4
{;}
O(1)
O(12)
O(123)
{N}
O(13)
(b) OD4 = OC4
{;}
O(1)
O(123)
{N}
O(13)
O(14) O(12)
(c) OV =h(12)(34),(13)(24)i
{;}
O(1)
O(123)
{N}
O(3)
O(134)
O(13)O(14)
O(12)O(34)
(d) OS24=h(12)(34)i
Figure 1: Indecomposable power set orbits on N = {1, 2, 3, 4}
Power Set Orbits under di↵erent groups are also ordered by refinement of associated
partition of the power set.
Power Set Orbits, N = 4
{;}
O(1)
O(12)
O(123)
{N}
(a) OS4 = OA4
{;}
O(1)
O(12)
O(123)
{N}
O(13)
(b) OD4 = OC4
{;}
O(1)
O(123)
{N}
O(13)
O(14) O(12)
(c) OV =h(12)(34),(13)(24)i
{;}
O(1)
O(123)
{N}
O(3)
O(134)
O(13)O(14)
O(12)O(34)
(d) OS24=h(12)(34)i
Figure 1: Indecomposable power set orbits on N = {1, 2, 3, 4}
Power Set Orbits under di↵erent groups are also ordered by refinement of associated
partition of the power set.
¯
⇤G
?=
G
– Complete Answer for N = 5
Symmetric, Alternating, & Gen. A�ne:
S5 : �(12345), (12)�, A5 : �(12345), (123)�GA(1, 5) : �(12345), (2345)�
Cyclic & Dihedral:
C5 : �(12345)�D5 : �(12345), (25)(34)�
(2, 3)-Partition:
S2 ⇥ S3 : �(12), (345), (34)�S2 ⇥ A3 : �(12), (345)�S3
5 : �(12)(45), (345)�
(1, 4)-Partition:
S1 ⇥ S4 : �(2345), (23)�S1 ⇥ A4 : �(2345), (23)(45)�
Fix 1, Rest Cyclic or Dihedral:
S1 ⇥ C4 : �(2345)�S1 ⇥ D4 : �(2345), (24)�
Fix 1, Rest Klein 4-group:
S1 ⇥ V : �(23)(45), (24)(35)�
(1, 1, 3)-partition:
S1 ⇥ S1 ⇥ S3 : �(345), (34)�S1 ⇥ S1 ⇥ A3 : �(345)�
(1, 2, 2)-partition:
S1 ⇥ S2 ⇥ S2 : �(23), (45)�
(1, 1, 1, 2)-partition:
S1 ⇥ S1 ⇥ S1 ⇥ S2 : �(45)�
trivial: �()�
S1 ⇥ S24 : �(23)(45)�
G
=
⇤G
G
=
⇤G
G
=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
Power Set Orbits, N = 5
{;}
O(1)
O(12)
O(123)
O(1234)
{N}
(a) OS5 = OA5 = OGA(1,5)
{;}
O(1)
O(12)
O(123)
O(1234)
{N}
O(13)
O(124)
(b) OD5 = OC5
Figure 2: Indecomposable orbit structures on N = {1, 2, 3, 4, 5}
Observe that while a poset, need not be a lattice.
¯
⇤G
?=
G
– Some Answers for N � 6
Thm. 5: For n � 6,
• ⇤Cn
( Cn ,
• ⇤Dn
( Dn ,
• ⇤S1⇥Cn�1
( S1⇥Cn�1 , &
• ⇤S1⇥Dn�1
( S1⇥Dn�1 .
Proof shows that the ray with coordinates matching Vamos projection lies in Cn but
is cut o↵ by Zhang-Yeung non-Shannon inequality.
Summary
Symmetric, Alternating, & Gen. A�ne:
S5 : �(12345), (12)�, A5 : �(12345), (123)�GA(1, 5) : �(12345), (2345)�
Cyclic & Dihedral:
C5 : �(12345)�D5 : �(12345), (25)(34)�
(2, 3)-Partition:
S2 ⇥ S3 : �(12), (345), (34)�S2 ⇥ A3 : �(12), (345)�S3
5 : �(12)(45), (345)�
(1, 4)-Partition:
S1 ⇥ S4 : �(2345), (23)�S1 ⇥ A4 : �(2345), (23)(45)�
Fix 1, Rest Cyclic or Dihedral:
S1 ⇥ C4 : �(2345)�S1 ⇥ D4 : �(2345), (24)�
Fix 1, Rest Klein 4-group:
S1 ⇥ V : �(23)(45), (24)(35)�
(1, 1, 3)-partition:
S1 ⇥ S1 ⇥ S3 : �(345), (34)�S1 ⇥ S1 ⇥ A3 : �(345)�
(1, 2, 2)-partition:
S1 ⇥ S2 ⇥ S2 : �(23), (45)�
(1, 1, 1, 2)-partition:
S1 ⇥ S1 ⇥ S1 ⇥ S2 : �(45)�
trivial: �()�
S1 ⇥ S24 : �(23)(45)�
G
=
⇤G
G
=
⇤G
G
=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
Symmetric & Alternating:
S4 : �(1234), (12)�A4 : �(123), (12)(34)�
Dihedral & Cyclic:
D4 : �(1234), (13)�C4 : �(1234)�
Normal Klein 4-group:
V : �(12)(34), (13)(24)�
Double Transp.:
S24 : �(12)(34)�
(1, 3)-Partition:
S1 ⇥ S3 : �(234), (23)�S1 ⇥ A3 : �(234), (243)�
(2, 2)-Partition:
S2 ⇥ S2 : �(12), (34)�
(1, 1, 2)-Partition:
S1 ⇥ S1 ⇥ S2 : �(34)�
Trivial: �()�
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
6=
⇤G
G
=
⇤G
G
=
⇤G
G
=
⇤G
G
=
⇤G
n = 4 :
n = 5 :
n � 6 :
Cn 6=
⇤Cn
,
Dn 6=
⇤Dn
{1, . . . , n}//G not a (1, n � 1) or (n) partition =)
G
6=
⇤G
S1⇥Cn�1 6=
⇤S1⇥Cn�1
,
S1⇥Dn�1 6=
⇤S1⇥Dn�1
Fig. 1(a)
Fig. 1(b)
Fig. 1(c)
Fig. 1(d)
Fig. 2(a)
Fig. 2(b)
Fig. 3
Sn =
⇤Sn
, S1⇥Sn =
⇤S1⇥Sn
The Way Forward
1. For n � 6, answer ¯
⇤G
?=
G
for other maximal transitive groups.
2. Permutations of the ground set form just one set of interesting symmetries for �N
,
others include
• Combinatorial Symmetry Group: permutations of the extreme rays which
leave the face lattice intact. (Huge – Subgroup of SM
, w/ M= # of extreme
rays)
• A�ne Symmetry Group: Those Combinatorial symmetries whose ray
permutation can be generated by multiplying by a 2
N � 1 ⇥ 2
N � 1 invertible
matrix. (Also a large group)
• Restricted A�ne Symmetry Group: Those a�ne symmetries associated with
vectors representing the rays of fixed length. (can be computed w/ sympol)
3. Which of the latter are also symmetries of
¯
�
⇤N
?
4. What sort of dimensionality reduction can be achieved by exploiting the a�ne
symmetries when calculating rate regions for network coding, storage repair
tradeo↵s, caching regions, etc?