Introduction to Programming

Message PassingLoopy BP and Message DecodingProbabilisticGraphicalModelsInference

Daphne Koller1Message Coding & DecodingU1, , UkEncoderX1, , XnNoisyChannelY1, , YnDecoderV1, , VkRate = k/nBit error rate =

Daphne KollerChannel CapacityNoisyChannel01010.90.90.10.101010.90.90.10.1?BinarysymmetricchannelBinaryerasurechannelcapacity = 0.531capacity = 0.90Xcapacity =

Daphne KollerShannons Theorem

McElieceRate in multiples of capacityBit error probabilityDaphne KollerHow close to C can we get?

Signal to noise ratio (dB)Log bit error probMcEliece-1-2-3-4-5-6Daphne KollerTurbocodes (May 1993)

Daphne Koller

How close to C can we get?Signal to noise ratio (dB)Log bit error prob.Shannon limit = -0.79McEliece-1-2-3-4-5-6-7Daphne KollerTurbocodes: The IdeaU1, , UkNoisyChannelEncoder 1Encoder 2Y11, , Y1nY21, , Y2nCompute P(U | Y1, Y2)Y11, , Y1nY21, , Y2nDecoder 1Decoder 2Bel(U | Y2)Bel(U | Y1)Daphne KollerIterations of Turbo Decoding

Signal to noise ratio (dB)Log bit error prob.-1-2-3-4-5-6Daphne KollerX7Y5Y7Y6X5X6U1U4U2X4X1X2Y4Y1Y2U3X3Y3Low-Density Parity Checking CodesInvented byRobert Gallagerin 1962!original message bitsparity bitsDaphne Koller10Decoding as Loopy BPX7Y5Y7Y6X5X6U1U4U2X4X1X2Y4Y1Y2U3X3Y3X5clusterX7clusterX6clusterU2U3U4U1Daphne KollerDecoding in Action

Information Theory, Inference, and Learning Algorithms, David MacKayDaphne KollerTurbo-Codes & LDPCs3G and 4G mobile telephony standardsMobile television system from QualcommDigital video broadcastingSatellite communication systemsNew NASA missions (e.g., Mars Orbiter)Wireless metropolitan network standardDaphne KollerSummaryLoopy BP rediscovered by coding practitionersUnderstanding turbocodes as loopy BP led to development of many new and better codesCurrent codes coming closer and closer to Shannon limitResurgence of interest in BP led to much deeper understanding of approximate inference in graphical modelsMany new algorithmsDaphne KollerEND END ENDDaphne Koller