HRIT Decoding (J2K and missing CPPDUs)

[kisaa]

I got a dump of an HRIT stream and am trying to decode it using xrit-rx. I get a lot of errors like

   [TP_File] COUNTER: 0 (Other, SEGMENT:?) LENGTH: 5078090
     LENGTH: ERROR (EXPECTED: 5078090, ACTUAL: 5069900, DIFF: -8190)
     SKIPPING FILE DUE TO DROPPED PACKETS
   NO CP_PDU TO FINISH (DROPPED PACKETS?)
   [CP_PDU] EOF MARKER

Product files (.bin) are written, but their size is too small.

I have checked VCDU counter field, and I does not see any packet loss from receiver, counter leaps only on transmission start/stop:

Time               Prev   Current
12:46:39 Counter leap: e1d486 0
12:51:50 Counter leap: 1bcf2  e1d487
12:56:39 Counter leap: e371e3 0
13:01:51 Counter leap: 1be25  e371e4
13:06:39 Counter leap: e50e09 0
13:11:52 Counter leap: 1bf18  e50e0a
13:16:39 Counter leap: e6a940 0
13:21:52 Counter leap: 1bfe4  e6a941
13:26:39 Counter leap: e843ab 0
13:31:53 Counter leap: 1c081  e843ac
13:36:40 Counter leap: e9de29 0
13:41:53 Counter leap: 1c0cf  e9de2a
13:46:40 Counter leap: eb77a9 0

Links to dumps:
hrit03.dmp
hrit03.log
hrit04.dmp
hrit04.log

[sam210723]

I haven't been able to test HRIT since goesrecv has not been modified for QPSK yet. What is your demodulator and decoder chain?

Thanks for the VCDU files, I'll take a look!

[sam210723]

HRIT headers are intact but the JPEG/JFIF headers are broken for some reason. There does seem to be a bigger demuxing issue so I'll take a look at that first, then look at the JPEG issues.

[kisaa]

What is your demodulator and decoder chain?

This is a professional meteorological receiver, we used it to get HRIT / LRIT with Electro-L. It supports Viterbi and Reed-Solomon decoding and provides 1024 (full frame with sync word), 1020 (CVCDU) or 892 (VCDU) bytes packets over Ethernet.
I changed the main script a bit to get VCDUs from UDP socket with the correct interleaving settings.
Unfortunately, my skills are not enough to write a QPSK software demodulator (
Maybe later I can adapt https://github.com/dbdexter-dev/meteor_demod for these needs - it is written for QPSK / OQPSK.

[sam210723]

Nice! is it a Dartcom receiver?

I have started work on a QPSK DSP chain using GNURadio source for reference. Using LRPT code as well is a good idea! I'll be getting back to that soon.

[kisaa]

is it a Dartcom receiver?

No, this is a Russian model produced by http://www.etra-plus.ru/ (D8L, could not be found on the site, possibly EOL)

[sam210723]

Interesting, I thought it looked similar to the Comtech CDM modem series, then I found this on the D4L product page:

(Comtech CDM-600)

(ETRA-D4L)

[sam210723]

The JPEG headers in these images seem to be malformed and don't have enough info to recover the image data.

They are basically empty when compared to a normal JPEG/JFIF header. I'm going through the HRIT documentation at the moment to see if there's anything explaining this.

[kisaa]

According to http://nmsc.kma.go.kr/resources/homepage/pdf/GK2A_HRIT_Mission_Specification_Documen_v1.0.pdf, p.24,
JPEG2000 is used, not regular JPEG. But header looks untypical even for JPEG2000.

[sam210723]

This makes sense as Pillow was throwing exceptions when loading the image. It needs OpenJPEG installed and in PATH to handle J2K.

It also mentions they're using J2K in a lossless configuration so that could explain the sparse headers.

[kisaa]

Got images from IR channels!
Decompressed HRIT data using jpeg utility from https://github.com/thorfdbg/libjpeg, with intermediate conversion to PPM and regular JPG and building all segments together, by (dirty, sorry for this!) calling hrit2jpg.processHRIT() from demuxer.py (Channel.handle_xRIT) after 10th segment is saved to disk:

        else:
            # Print XRIT file info
            xrit.print_info()
            if xrit.FILE_NAME.endswith('10.hrit'):
                hrit2jpg.proceedHRIT(xrit.get_save_path(self.config.output))

Still having troubles with VIS channel ((

hrit2jpg.zip

[sam210723]

Nice work! I'll look at streamlining that process then integrate it into xrit-rx.

[kisaa]

I left the system running overnight, and today I discovered the following:
sometimes (I have no idea about the pattern) one or two segments of the VIS channel are saved successfully, and then the image is successfully (partly) decoded.
The rest of the time, I get the following message from the handle_CPPDU function (I changed the line if self.config.verbose: to if True: to see only these debug messages):

[VCID 0] GK-2A: FULL DISK
  [XRIT] "IMG_FD_038_IR105_20200602_061636_01.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_01.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_01.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_01.hrit"
    LENGTH:     ERROR (EXPECTED: 5282322, ACTUAL: 5274132, DIFF: -8190)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_WV069_20200602_061636_02.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_02.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_02.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_02.hrit"
    LENGTH:     ERROR (EXPECTED: 8108234, ACTUAL: 8091854, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR105_20200602_061636_03.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_03.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_03.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_03.hrit"
    LENGTH:     ERROR (EXPECTED: 9413834, ACTUAL: 9397454, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR105_20200602_061636_04.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_04.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_04.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_04.hrit"
    LENGTH:     ERROR (EXPECTED: 9086738, ACTUAL: 9070358, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR105_20200602_061636_05.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_05.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_05.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_05.hrit"
    LENGTH:     ERROR (EXPECTED: 9166682, ACTUAL: 9150302, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_WV069_20200602_061636_06.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_06.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_06.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_06.hrit"
    LENGTH:     ERROR (EXPECTED: 9326610, ACTUAL: 9310230, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_SW038_20200602_061636_07.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_07.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_07.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_07.hrit"
    LENGTH:     ERROR (EXPECTED: 8890946, ACTUAL: 8874566, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR123_20200602_061636_08.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_08.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_08.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_08.hrit"
    LENGTH:     ERROR (EXPECTED: 7802050, ACTUAL: 7785670, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_WV069_20200602_061636_09.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_09.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_09.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_09.hrit"
    LENGTH:     ERROR (EXPECTED: 7175426, ACTUAL: 7167236, DIFF: -8190)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR123_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_IR123_20200602_061636.jpg"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_IR105_20200602_061636.jpg"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_WV069_20200602_061636.jpg"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_SW038_20200602_061636.jpg"
  [XRIT] "IMG_FD_038_VI006_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_VI006_20200602_061636.jpg"

[VCID 63] GK-2A: IDLE

(lines with [JPEG] in last section is output of my script).

It is suspicious that the number of missing bytes is the same in all segments (2 CP_PDU) except the first and last, where it is one CP_PDU.
Upd: for some sessions, diff can be -8190 for other segments too

[kisaa]

New info about CPPDU loss:

after adding CPPDU counter to TP_file and checking its value in Demuxer.Channel.handle_CPPDU, have this log:

[VCID 0] GK-2A: FULL DISK
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 30
  [XRIT] "IMG_FD_026_SW038_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 31
    CPPDU COUNTER:    LAST# 54
  [XRIT] "IMG_FD_026_WV069_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 55
    CPPDU COUNTER:    LAST# 98
  [XRIT] "IMG_FD_026_IR105_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 42
  [XRIT] "IMG_FD_026_IR123_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER (C): ERROR (EXPECTED: 441, ACTUAL: 442)
    CPPDU COUNTER:    LAST# 672
  [XRIT] "IMG_FD_026_VI006_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 99
    CPPDU COUNTER:    LAST# 143
  [XRIT] "IMG_FD_026_SW038_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 673
    CPPDU COUNTER:    LAST# 739
  [XRIT] "IMG_FD_026_IR105_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 43
    CPPDU COUNTER:    LAST# 79
  [XRIT] "IMG_FD_026_WV069_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 80
    CPPDU COUNTER:    LAST# 145
  [XRIT] "IMG_FD_026_IR123_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER (C): ERROR (EXPECTED: 441, ACTUAL: 442)
    CPPDU COUNTER (C): ERROR (EXPECTED: 883, ACTUAL: 884)
    CPPDU COUNTER:    LAST# 1019
  [XRIT] "IMG_FD_026_VI006_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 144
    CPPDU COUNTER:    LAST# 188
  [XRIT] "IMG_FD_026_SW038_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 740
    CPPDU COUNTER:    LAST# 781
  [XRIT] "IMG_FD_026_WV069_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 189
    CPPDU COUNTER:    LAST# 258
  [XRIT] "IMG_FD_026_IR123_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 782
    CPPDU COUNTER:    LAST# 854
  [XRIT] "IMG_FD_026_IR105_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 855
    CPPDU COUNTER (C): ERROR (EXPECTED: 1296, ACTUAL: 1297)
    CPPDU COUNTER (C): ERROR (EXPECTED: 1738, ACTUAL: 1739)
    CPPDU COUNTER:    LAST# 2029
  [XRIT] "IMG_FD_026_VI006_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 70
  [XRIT] "IMG_FD_026_IR105_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 146
    CPPDU COUNTER:    LAST# 188
  [XRIT] "IMG_FD_026_SW038_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 189
    CPPDU COUNTER:    LAST# 255
  [XRIT] "IMG_FD_026_IR123_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 256
    CPPDU COUNTER:    LAST# 294
  [XRIT] "IMG_FD_026_WV069_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 1020
    CPPDU COUNTER (C): ERROR (EXPECTED: 1461, ACTUAL: 1462)
    CPPDU COUNTER (C): ERROR (EXPECTED: 1903, ACTUAL: 1904)
    CPPDU COUNTER:    LAST# 2210
  [XRIT] "IMG_FD_026_VI006_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 259
    CPPDU COUNTER:    LAST# 335
  [XRIT] "IMG_FD_026_IR123_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 2211
    CPPDU COUNTER:    LAST# 2290
  [XRIT] "IMG_FD_026_IR105_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 336
    CPPDU COUNTER:    LAST# 381
  [XRIT] "IMG_FD_026_SW038_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 71
    CPPDU COUNTER:    LAST# 118
  [XRIT] "IMG_FD_026_WV069_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 2030
    CPPDU COUNTER (C): ERROR (EXPECTED: 2471, ACTUAL: 2472)
    CPPDU COUNTER (C): ERROR (EXPECTED: 2913, ACTUAL: 2914)
    CPPDU COUNTER:    LAST# 3286
  [XRIT] "IMG_FD_026_VI006_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 382
    CPPDU COUNTER:    LAST# 425
  [XRIT] "IMG_FD_026_WV069_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 3287
    CPPDU COUNTER:    LAST# 3364
  [XRIT] "IMG_FD_026_IR105_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 3365
    CPPDU COUNTER:    LAST# 3439
  [XRIT] "IMG_FD_026_IR123_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 119
    CPPDU COUNTER:    LAST# 163
  [XRIT] "IMG_FD_026_SW038_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 295
    CPPDU COUNTER (C): ERROR (EXPECTED: 736, ACTUAL: 737)
    CPPDU COUNTER (C): ERROR (EXPECTED: 1178, ACTUAL: 1179)
    CPPDU COUNTER:    LAST# 1561
  [XRIT] "IMG_FD_026_VI006_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 1562
    CPPDU COUNTER:    LAST# 1605
  [XRIT] "IMG_FD_026_SW038_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 2291
    CPPDU COUNTER:    LAST# 2330
  [XRIT] "IMG_FD_026_WV069_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 2331
    CPPDU COUNTER:    LAST# 2400
  [XRIT] "IMG_FD_026_IR123_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 426
    CPPDU COUNTER:    LAST# 499
  [XRIT] "IMG_FD_026_IR105_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 2401
    CPPDU COUNTER (C): ERROR (EXPECTED: 2842, ACTUAL: 2843)
    CPPDU COUNTER (C): ERROR (EXPECTED: 3284, ACTUAL: 3285)
    CPPDU COUNTER:    LAST# 3615
  [XRIT] "IMG_FD_026_VI006_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 500
    CPPDU COUNTER:    LAST# 541
  [XRIT] "IMG_FD_026_SW038_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 3440
    CPPDU COUNTER:    LAST# 3508
  [XRIT] "IMG_FD_026_IR105_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 164
    CPPDU COUNTER:    LAST# 193
  [XRIT] "IMG_FD_026_WV069_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 542
    CPPDU COUNTER:    LAST# 607
  [XRIT] "IMG_FD_026_IR123_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 3509
    CPPDU COUNTER (C): ERROR (EXPECTED: 3950, ACTUAL: 3951)
    CPPDU COUNTER (C): ERROR (EXPECTED: 4392, ACTUAL: 4393)
    CPPDU COUNTER:    LAST# 4617
  [XRIT] "IMG_FD_026_VI006_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 4618
    CPPDU COUNTER:    LAST# 4682
  [XRIT] "IMG_FD_026_IR105_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 1606
    CPPDU COUNTER:    LAST# 1641
  [XRIT] "IMG_FD_026_SW038_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 608
    CPPDU COUNTER:    LAST# 670
  [XRIT] "IMG_FD_026_IR123_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 1642
    CPPDU COUNTER:    LAST# 1672
  [XRIT] "IMG_FD_026_WV069_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 3616
    CPPDU COUNTER (C): ERROR (EXPECTED: 4057, ACTUAL: 4058)
    CPPDU COUNTER (C): ERROR (EXPECTED: 4499, ACTUAL: 4500)
    CPPDU COUNTER:    LAST# 4596
  [XRIT] "IMG_FD_026_VI006_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 4597
    CPPDU COUNTER:    LAST# 4618
  [XRIT] "IMG_FD_026_WV069_20200605_041636_10.hrit"
  [JPEG] "IMG_FD_026_WV069_20200605_041636.jpg"
    CPPDU COUNTER:    FIRST# 4619
    CPPDU COUNTER:    LAST# 4639
  [XRIT] "IMG_FD_026_SW038_20200605_041636_10.hrit"
  [JPEG] "IMG_FD_026_SW038_20200605_041636.jpg"
    CPPDU COUNTER:    FIRST# 671
    CPPDU COUNTER:    LAST# 709
  [XRIT] "IMG_FD_026_IR105_20200605_041636_10.hrit"
  [JPEG] "IMG_FD_026_IR105_20200605_041636.jpg"
    CPPDU COUNTER:    FIRST# 710
    CPPDU COUNTER:    LAST# 747
  [XRIT] "IMG_FD_026_IR123_20200605_041636_10.hrit"
  [JPEG] "IMG_FD_026_IR123_20200605_041636.jpg"
    CPPDU COUNTER:    FIRST# 4683
    CPPDU COUNTER (C): ERROR (EXPECTED: 5124, ACTUAL: 5125)
    CPPDU COUNTER:    LAST# 5162
  [XRIT] "IMG_FD_026_VI006_20200605_041636_10.hrit"

As you can see, every 442th from beginning of the TPFile CPPDU is loss. So, we have loss 8190 bytes for each 3619980 bytes (8190*442) of file length. For large file, there will be data loss after 7239960, 10859940 bytes and so on. As IR-channel's hrit files are much less than 3.6M, they have no data loss at all.