Memory violation error compiling in Delphi 10.2

[ruben-monmariola]

I use many features of FibPlus. I have tried to migrate to Delphi 10.2 and it has been impossible. Simple Datamodules with TpIBQuery work correctly. Datamodules with TpIBDataSet that use OnValidate or OnChange in most cases I get memory violation errors. I have tried to debug the program but the errors do not make sense, but they always occur inside FIBPlus functions. There is not a single line of code where it fails. Sometimes it is in an AsString, calculating the pointer where to save the data in the TFIBStringField, other times it is when calling Open, accessing an internal TList of the TpFIBDataset component, sometimes it is in an OnCalcFields, in the internal DataEvent. The only forum I have seen where there are other people with the same problem is https://www.clubdelphi.com/foros/showthread.php?t=93687

[did-vmonroig]

You're not alone. I've been monitorizing that thread and tried to apply some workarounds from there, but with no luck. Something really weird is happening when using latest Delphi versions with current Fibplus. I've spent several hours with no luck. Sorry,

I think it's related to issue #77 and PR #76.

[did-vmonroig]

@ruben-monmariola, if your Delphi version supports it, try disabling ASLR in linker options in both the Fibplus packages and your application project:

[ruben-monmariola]

Hi. I spent many hours trying to find something that would make sense of these errors and reviewing how FIBPlus allocates and frees memory, but I didn't find anything strange. I don't have those options that you mention in "linking". I'm still working with 32-bit versions. If you disable those options, does it no longer show the memory violation errors in 64-bit?

[did-vmonroig]

Disabling ASLR related options made a difference in 32-bit. For sure the second one made no effect as it's for 64-bit only, but the first one removed some random access violation errors, mainly in OnValidate and OnChange, but unfornately not all of them. I'm using Delphi 12.1 Athens.

Idea was found in this thread: https://en.delphipraxis.net/topic/6627-legacy-bad-code-issues-on-delphi-111-on-64bit-vcl-apps-related-to-new-aslr-linker-flags/. Later I found that there is a related issue, #60, also by @davideangeli, so maybe he could help us.

Right now I've one repeteable, in TFIBQuery.PreprocessSQL, line 5104, SetLength(vBeginParamsPos,PCount) that generates an error in System.pas line 36858, ReallocMem(pp, neededSize), with pp pointer being nil.

This is too weird and low level for me to debug as seems like internals of Delphi are failing with this package. I'm going to try other compiler and linker options. Of course, any thoughts are welcome.

[davideangeli]

Hi @did-vmonroig the issue #60 seems not closed but the changes that I made to solve it are already included in latest versions.

The problem that you are experiencing in TFIBQuery.PreprocessSQL could they be related to the SQL query that you are running? Too complex, strange characters?

Now in my applications I'm using fibPlus with Delphi 12.2 both 32bit and 64 bit connected to Firebird 5 databases and it is working pretty good. I've just done a couple of changes that I'll pull in a merge request but nothing so critical.

[did-vmonroig]

Thanks for your help, @davideangeli. The query is as simple as this:

SELECT MAX(NumeroOR)
  FROM OrdenesReparacion
  WHERE CodEmpresa = :CodEmpresa AND Ejercicio = :Ejercicio
    AND Serie = :Serie

Other more complicated queries run without this problem, but this is just an example. There is some kind of randomness in exceptions, difficult to reproduce them. So weird...

Could be possible to take a look to your .dpk, .dpr an .dproj files? I still think there could be some compiler option differences, something related to pointers or record alignment, introducing these problems. Thanks in advance.

[davideangeli]

@did-vmonroig if I simulate a similar query on my environment it works fine both 32/64 bits... To make some debug could be usefull a simple test case if reproducible.

I compile the fibPlus units in a custom runtime package. If could help these are the bpl 64 bit compiling options:

[did-vmonroig]

@davideangeli, thanks for providing your compiling options. There are some differences, like record field alignment or stack frames, but generating new DCU and BPL with your options made no difference.

I'm trying to extract the simplest reproducible sample that always throws an exception. It's not easy as exceptions are thrown with some extrange randomness and there are some third party components involved. In fact, I've started using FastMM with full debug and now exceptions are thrown in different conditions and involve stack corruption. For example:

Maybe this can lead to a new clue to more experienced developers.

[did-vmonroig]

I've tried record field alingment in all its flavours, altough in FIBplus.inc file, included in all .pas, there is a {$A+}, so BPL setting is not being applied. With it disabled or in less than Word, access violations are present in the very same moment of connection, so behavious is even worse.

@ruben-monmariola, may I know the third party components included in the project where you're using FibPlus? I would like to make a common list because our compiler version is not the same but the problems with random access violations are. If you prefer not disclosing that information here, we could talk about this privately.

[ruben-monmariola]

Mainly DevExpress and Indy. I also use madexcept. I tried disabling madexcept but the errors were the same.

[did-vmonroig]

With help from FastMM4 and FullDebugModeScanMemoryPoolBeforeEveryOperation option I think I'm closer to the reason for this access violations. At least I have a repeteable error, with this log:

---------------------------
Project.exe: Memory Error Detected
---------------------------
FastMM has detected an error during a free block scan operation. The block footer has been corrupted. 

The previous block size was: 2

This block was previously allocated by thread 0x3EDC, and the stack trace (return addresses) at the time was:

0040733A [System.pas][System][GetMem][4962]
00763B00 [FIBDataSet.pas][FIBDataSet][TFIBStringField.GetDataToReserveBuffer][1793]
00763B57 [FIBDataSet.pas][FIBDataSet][TFIBStringField.InternalGetAsString][1811]
00763BAE [FIBDataSet.pas][FIBDataSet][TFIBStringField.GetAsString][1836]
017FAE60 [DMod.pas][DMod][TDM.MyDataSetCalcFields][344]
006DE873 [Data.DB.pas][Data.DB][Db.TDataSet.DoOnCalcFields][15306]
0072A19D [pFIBDataSet.pas][pFIBDataSet][TpFIBDataSet.DoOnCalcFields][1231]
006DCE68 [Data.DB.pas][Data.DB][Db.TDataSet.CalculateFields][14300]
006DCDCE [Data.DB.pas][Data.DB][Db.TDataSet.GetCalcFields][14258]
00777881 [FIBDataSet.pas][FIBDataSet][TFIBCustomDataSet.GetCalcFields][9384]
007781F5 [FIBDataSet.pas][FIBDataSet][TFIBCustomDataSet.GetRecord][9724]

The allocation number was: 978767

The block was previously freed by thread 0x3EDC, and the stack trace (return addresses) at the time was:

007956D2 [fib.pas][fib][FIBAlloc$qqrpvuiui][372]
007509DE [FIBQuery.pas][FIBQuery][TFIBXSQLDA.Destroy][2649]
0040A4C7 [System.pas][System][TObject.Free][18403]
00756DA8 [FIBQuery.pas][FIBQuery][TFIBQuery.PreprocessSQL][5452]
007573CE [FIBQuery.pas][FIBQuery][TFIBQuery.Prepare][5597]
007541CC [FIBQuery.pas][FIBQuery][TFIBQuery.ExecQuery][4207]
0077B16B [FIBDataSet.pas][FIBDataSet][TFIBCustomDataSet.InternalOpen][11038]
00729C76 [pFIBDataSet.pas][pFIBDataSet][TpFIBDataSet.InternalOpen][1029]
006D9D0C [Data.DB.pas][Data.DB][Db.TDataSet.DoInternalOpen][12726]
006D9DBB [Data.DB.pas][Data.DB][Db.TDataSet.OpenCursor][12755]
006D9C6F [Data.DB.pas][Data.DB][Db.TDataSet.SetActive][12707]

The current thread ID is 0x3EDC, and the stack trace (return addresses) leading to this error is:

0041F6D6 [FastMM4][InternalScanMemoryPool$qqruiui]
0041F7B5 [FastMM4][ScanMemoryPoolForCorruptions$qqrv]
0041F13F [FastMM4][DebugGetMem$qqri]
0040733A [System.pas][System][GetMem][4962]
0040C33B [System.pas][System][NewUnicodeString][26043]
0040C57C [System.pas][System][UStrFromPWCharLen][26657]
0040C653 [System.pas][System][InternalUStrFromPCharLen$qqrr20System.UnicodeStringpcii][26834]
7C4C9D4C [Unknown function at fb_shutdown]
7C44FE3F [Unknown function at isc_blob_set_desc]
7C51116A [Unknown function at fb_shutdown]
7C511187 [Unknown function at fb_shutdown]

Current memory dump of 256 bytes starting at pointer address 7E4A1F70:

41 00 F7 4E 55 76 E4 01 80 0F E4 01 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 59 F7 0F 00 3A 73 40 00 C5 56 79 00 B6 16 75 00 CC 79 75 00 56 95 72 00 F0 96 EF 76
1D AF 77 00 76 9C 72 00 0C 9D 6D 00 BB 9D 6D 00 6F 9C 6D 00 DC 3E 00 00 DC 3E 00 00 D2 56 79 00
D2 09 75 00 C7 A4 40 00 FE 1A 75 00 C7 A4 40 00 D9 21 4E 00 60 1C 4E 00 9D 94 6D 00 2E 77 76 00
AA 8B 72 00 C7 A4 40 00 0E 00 00 00 08 00 45 46 5C F7 2F 44 8C 0C E4 01 80 0F E4 01 80 0F E4 01
80 0F A3 08 D0 BB E4 01 80 0F E4 01 00 00 00 00 C0 63 88 7E 00 00 00 00 00 00 00 00 28 F1 41 00
00 00 00 00 FF C1 0F 00 3A 73 40 00 B3 A3 40 00 B6 AA 40 00 98 A4 40 00 B6 AA 40 00 F3 A4 90 00
1B B2 A1 00 87 6E 00 01 4E 6F 00 01 9D 56 00 01 52 75 08 01 DC 3E 00 00 DC 3E 00 00 56 73 40 00

A  .  ÷  N  U  v  ä  .  €  .  ä  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  Y  ÷  .  .  :  s  @  .  Å  V  y  .  ¶  .  u  .  Ì  y  u  .  V  •  r  .  ð  –  ï  v
.  ¯  w  .  v  œ  r  .  .  �  m  .  »  �  m  .  o  œ  m  .  Ü  >  .  .  Ü  >  .  .  Ò  V  y  .
Ò  .  u  .  Ç  ¤  @  .  þ  .  u  .  Ç  ¤  @  .  Ù  !  N  .  `  .  N  .  �  ”  m  .  .  w  v  .
ª  ‹  r  .  Ç  ¤  @  .  .  .  .  .  .  .  E  F  \  ÷  /  D  Œ  .  ä  .  €  .  ä  .  €  .  ä  .
€  .  £  .  Ð  »  ä  .  €  .  ä  .  .  .  .  .  À  c  ˆ  ~  .  .  .  .  .  .  .  .  (  ñ  A  .
.  .  .  .  ÿ  Á  .  .  :  s  @  .  ³  £  @  .  ¶  ª  @  .  ˜  ¤  @  .  ¶  ª  @  .  ó  ¤  �  .
.  ²  ¡  .  ‡  n  .  .  N  o  .  .  �  V  .  .  R  u  .  .  Ü  >  .  .  Ü  >  .  .  V  s  @  .

Seems that in OnCalcFields event I could have a calculated field with TFIBStringField class, but maybe it should be TStringField. I'm going to try changing them in DFM. @ruben-monmariola, could you review too if you have any of this kind?

[did-vmonroig]

With ScanMemoryPoolForCorruptions from FastMM4 calls I've managed to find where the error is really throwing. When a call to AsString is made inside OnValidate events for the validated field, string buffer is reallocated by GetDataToReserveBuffer, corrupting heap and throwing exceptions later, when freeing objects.

I can't understand why few people finds this problem. Maybe is because Win32 implementation differs from Win64, or they are not using OnValidate the way I do. @ruben-monmariola, do you use persistent fields with OnValidate and AsString calls to the field being validated?

I'm testing a patch that solves in my end this problem and also #78. If it's stable I'll send a PR.