Heavy load with monster-lists kills Bricscad

Peter2

I use a DosLib command to read XYZ coordinates from very long text files and to save it to Autolisp-lists; up to 120MB with 4.000.000 lines.
Autocad 2018 on same machine with same data and same DoslLib version handles the maximal load, Bricscad handles "only" 30MB with 1.000.000 lines, then it freezes with full processor load and stable RAM usage. here is the log-file, the last event was not written because Bricscad was already frozen

: (progn (gc) (setq test nil test (dos_readtextfile "d:\\Downloads\\GD201113-087-R\\1mb.xyz"))(princ))
: (progn (setq test nil test (dos_readtextfile "d:\\Downloads\\GD201113-087-R\\30mb.xyz"))(princ))
: (length test)
999999
: (progn (setq test nil test (dos_readtextfile "d:\\Downloads\\GD201113-087-R\\40mb.xyz"))(princ))

; ----- LISP : Call Stack -----
; [0]...DOS_READTEXTFILE <<--
;
; ----- Error around expression -----
; (AL_EXECUTEADSDEFUN "DOS_READTEXTFILE" REST)
; in file : 
; v:\Bricsys\Support\DOSLib20x64.brx
;
; error : out of LISP 'Heap' memory at [gc]
: (progn (setq test nil test (dos_readtextfile "d:\\Downloads\\GD201113-087-R\\1mb.xyz"))(princ))
: (length test)
40001
: (progn (setq test nil test (dos_readtextfile "d:\\Downloads\\GD201113-087-R\\30mb.xyz"))(princ))
: (length test)

; ----- Error around expression -----

The usage of (GC) improves it not really.
Any ideas to handle it?

James Maeding

yep, switch to this function:
;(princ (car (FILE-TO-LST-ALL "C:\\Users\\james\\Desktop\\CALC ARC TABLE.cgt"))) (defun FILE-TO-LST-ALL (Fil / FilObj FilPth FilSys OpnFil RetVal FileH TextString) (if (setq FilPth (findfile Fil)) (progn (setq FilSys (vlax-create-object "Scripting.FileSystemObject") FilObj (vlax-invoke FilSys "GetFile" FilPth) OpnFil (vlax-invoke FilObj "OpenAsTextStream" 1 0) ) (while (= (vlax-get OpnFil "AtEndOfStream") 0) (setq RetVal (cons (vlax-invoke OpnFil "ReadLine") RetVal)) ) (vlax-invoke OpnFil "Close") (vlax-release-object OpnFil) (vlax-release-object FilObj) (vlax-release-object FilSys) (setq RetVal (reverse RetVal)) (if (and (> (length RetVal) 0) (> (strlen (car RetVal)) 2) (= (substr (car RetVal) 1 3) "") ) (setq RetVal (cons (substr (car RetVal) 4)(cdr RetVal))) ) RetVal ) nil ) )

I've used it so long I don't even recall what that "" thing is all about.
I researched this quite a bit so think its at least as good as the doslib approach, in speed and robustness.
Not sure if a mac has the same Scripting.FileSystemObject, or other OS's's's'. I'm on windows.

James Maeding

BTW, no point in setting test to nil there first:
(progn (gc) (setq test nil test (dos_readtextfile "d:\Downloads\GD201113-087-R\1mb.xyz"))(princ))
Even that GC will not help, in my experience.
I'm guessing if the doslib function fails, it either gives nil, or crashes the lisp.
In either case, you are not going to care about previous value of test being obliterated.
Subtle details, but I see people clutter code with extra stuff and it just confuses things.

Torsten Moses

Dear Peter2,

actually, our LISP engine uses a "fixed" memory range (usually ~200 MB) ... so indeed, huge lists can cause out-of-memory conditions ...
3 solutions :
a) there is lispex.dll.cfg where you can define memory limits (actually there is a bug : the value for intended "VM_MAXIMUM_MEM" needs to be set at "VM_PAGE_OVER_HEAP" parameter; already fixed internally, also memory preset increases to 256 MB; both planned for V21.2)
b) use a bit smarter Lisp code - especially try to avoid storage of the same huge data with multiple global variables
(global variables are never cleared by GarbageCollection, so eventually set the global variable to NIL, when appropriate)
c) you might use (vle-file->list filename commentchar) which returns all file lines as a list (could be more memory-efficient than via DOSLib)

Also, you might run (mem) at commandline, to see the LISP memory usage (whether it is full)

hope this explains & helps a bit ?

Peter2

It find that the background is clear now.
In the meantime I replaced the way "read entire file to list and check list" with "read one line, check the data, write the result to a (much) smaller list". It works, but needs time - approx. 10-15000 values per second.

@James Maeding
I just started a short posting relating to GC at the Swamp.
http://www.theswamp.org/index.php?topic=56381.0

@Torsten Moses
b) my real code reads and stores the data only once, and in my example above I set the variable to nil.
Or do you mean something else?

I will check your code examples, thanks.

Edit:
Just for documenatation -

• reading of 30 MB needs 95% of available space
• (setq test nil) reduces no mem

• (GC) cleans the mem

  (mem)
  GC calls Total 0 [0.00 secs.] (0 in all docs)
  GC calls System 0, Int 0, Real 0, Vector 0, Strg 0, Symbol 0, Cons 0, Heap 0

  VM total memory : 199.996 MB
  VM Page memory : 120.000 MB available, 19.625 MB used (16 %)
  VM Heap memory : 79.996 MB available, 0.844 MB used (1 %)

  nil

  (progn (setq test nil test (dos_readtextfile "d:\Downloads\GD201113-087-R\30mb.xyz"))(princ))

  (mem)

  GC calls Total 244 [2.79 secs.] (244 in all docs)
  GC calls System 0, Int 0, Real 0, Vector 0, Strg 3, Symbol 0, Cons 0, Heap 241

  VM total memory : 199.996 MB
  VM Page memory : 120.000 MB available, 53.875 MB used (44 %)
  VM Heap memory : 79.996 MB available, 76.741 MB used (95 %)

  nil

  (setq test nil)
  nil

  (mem)

  GC calls Total 244 [2.79 secs.] (244 in all docs)
  GC calls System 0, Int 0, Real 0, Vector 0, Strg 3, Symbol 0, Cons 0, Heap 241

  VM total memory : 199.996 MB
  VM Page memory : 120.000 MB available, 53.875 MB used (44 %)
  VM Heap memory : 79.996 MB available, 76.741 MB used (95 %)

  nil

  (gc)
  nil

  (mem)

  GC calls Total 245 [2.81 secs.] (245 in all docs)
  GC calls System 1, Int 0, Real 0, Vector 0, Strg 3, Symbol 0, Cons 0, Heap 241

  VM total memory : 199.996 MB
  VM Page memory : 120.000 MB available, 53.875 MB used (44 %)
  VM Heap memory : 79.996 MB available, 0.447 MB used (0 %)

And (vle-file->list ..) can not handle the 30MB file:

: (vle-file->list "d:\\Downloads\\GD201113-087-R\\mini.txt" nil)
("a a" "b b")

: (progn (setq test (vle-file->list "d:\\Downloads\\GD201113-087-R\\30mb.xyz" nil)) (princ))

; ----- Error around expression -----
; (VLE-FILE->LIST "d:\\Downloads\\GD201113-087-R\\30mb.xyz" NIL)
;
: (mem)

GC calls Total  427 [10.46 secs.]  (429 in all docs)
GC calls System 1, Int 0, Real 0, Vector 0, Strg 3, Symbol 0, Cons 0, Heap 425

VM total memory : 199.996 MB
VM Page  memory :  77.500 MB available,  77.500 MB used (100 %)
VM Heap  memory : 122.496 MB available, 122.496 MB used (99 %)

nil
: (setq test nil)
nil
: (mem)

GC calls Total  427 [10.46 secs.]  (429 in all docs)
GC calls System 1, Int 0, Real 0, Vector 0, Strg 3, Symbol 0, Cons 0, Heap 425

VM total memory : 199.996 MB
VM Page  memory :  77.500 MB available,  77.500 MB used (100 %)
VM Heap  memory : 122.496 MB available, 122.496 MB used (99 %)

nil
: (gc)

; ----- Error around expression -----
; (GC)
;
; error : out of LISP 'Heap' memory at [gc]

Torsten Moses

Dear Peter,

did you use
(vle-file->list "d:\Downloads\GD201113-087-R\mini.txt" nil)
in a new drawing ? As the previous run with DOSLib might already keep the memory filled ?

As mentioned, will get better with V21.2 ... more memory available;
besides, the memory setup via " lispex.dll.cfg" is still possible (under wrong key today, corrected with V21,2 as well)

many greetings !

Peter2

did you use
(vle-file->list "d:\Downloads\GD201113-087-R\mini.txt" nil)
in a new drawing ? As the previous run with DOSLib might already keep the memory filled ?

Yes. Same problem

I tried to modify the "wrong" settings (20.2 German) and had before:

Setting:
VM_MAXIMUM_MEM # 0
VM_PAGE_OVER_HEAP # 0
MDI_LISP_COMPRESS # 0

Result:
: (mem)
GC calls Total 0 [0.00 secs.] (0 in all docs)
GC calls System 0, Int 0, Real 0, Vector 0, Strg 0, Symbol 0, Cons 0, Heap 0
VM total memory : 199.996 MB
VM Page memory : 120.000 MB available, 19.625 MB used (16 %)
VM Heap memory : 79.996 MB available, 0.844 MB used (1 %)

new Settings
VM_MAXIMUM_MEM # 0
VM_PAGE_OVER_HEAP 555
or
VM_MAXIMUM_MEM 50
VM_PAGE_OVER_HEAP 555
or
VM_MAXIMUM_MEM 555
VM_PAGE_OVER_HEAP 50
or
VM_MAXIMUM_MEM 555
VM_PAGE_OVER_HEAP # 0

and restarted Brciscad - and nothing changed

Torsten Moses

Dear Peter,

I just tried on V20.2 version :
VM_MAXIMUM_MEM # 0
VM_PAGE_OVER_HEAP # 400
=>
(mem)
VM total memory : 399.996 MB
VM Page memory : 240.000 MB available, 19.625 MB used (8 %)
VM Heap memory : 159.996 MB available, 0.342 MB used (0 %)

and with
VM_MAXIMUM_MEM # 0
VM_PAGE_OVER_HEAP # 600
=>
VM total memory : 599.996 MB
VM Page memory : 360.000 MB available, 19.625 MB used (5 %)
VM Heap memory : 239.996 MB available, 0.342 MB used (0 %)

So that should be sufficient to run your code then ?
many greetings !

James Maeding

oh, an .xyz involved. You are dealing with GIS level data, not light for sure.
@Torsten Moses
Is that 250 mb memory limit similar or different from acad? I did not know there was a hard total memory limit on lisp vars.
So nice to have you commenting on this to get reliable info.

I will add that with heavy data, you will absolutely want the fastest read and manipulate code you can use.
I switch to .net in these cases. I read in 100 million pts from a .laz point cloud file, possibly where that .xyz came from too.
That learning curve is medium hard IMO, with someone like me explaining how to set up a test project and getting you going in right direction.
Its hard if you just try on your own. The code is easy, its the understanding of how visual studio organizes things that needs guidance.
You can make a lisp command using .net, and I replaced doslib for myself that way. Its always better if you have code an full control.
thx

Terry Dotson

@James Maeding said:
I switch to .net in these cases ...

If you hit the default 2GB limit of .NET, you can override that with the gcAllowVeryLargeObjects addition to the .config file. I've had MASSIVE lidar contents entirely in memory. I just wish CAD engines didn't choke on them.

Here's a blog post on the details.

James Maeding

@Terry Dotson
Funny, I've never hit that and I load .laz files that make my RAM usage go up by 10 gb at least.
I thought windows 64 bit got around the 2 gb.

Terry Dotson

@James Maeding said:
Funny, I've never hit that and I load .laz files that make my RAM usage go up by 10 gb at least.
I thought windows 64 bit got around the 2 gb.

It's only available on 4.5 and 64bit. Here's the Microsoft details.

James Maeding

@Terry Dotson
Your links are not showing, do you see them? I'm running firefox.

Terry Dotson

Firefox here also, hover over them? I'm just typing words like "Microsoft details", swiping across those words and using this forums URL icon, pasting the address.

James Maeding

here is what I see, no link...

ScottS

The links are there, but look identical to regular text. Persistent flaw in this forum.

James Maeding

oh, I see now. Wow, like an easter egg! thanks @ScottS !

Peter2

Good morning Torsten

I just tried on V20.2 version :
VM_MAXIMUM_MEM # 0
VM_PAGE_OVER_HEAP # 400
=>
(mem)
VM total memory : 399.996 MB
VM Page memory : 240.000 MB available, 19.625 MB used (8 %)
VM Heap memory : 159.996 MB available, 0.342 MB used (0 %)

I see that I misunderstood the information and removed the # from the definition. Now with the right definition it expands the memory as expected.

But nevertheless there are 2 problems:
a)
This is a modification on the local machine, needing admin rights. It is complicated or impossible for usage on other machines or foreign machines

b)
Reading a 40MB file needs 135 MB memory, 50MB is too much for the settings above. So my rude way to read one-per-one-line is more stable then creating monster lists

    : (mem)

    GC calls Total  0 [0.00 secs.]  (0 in all docs)
    GC calls System 0, Int 0, Real 0, Vector 0, Strg 0, Symbol 0, Cons 0, Heap 0

    VM total memory : 399.996 MB
    VM Page  memory : 240.000 MB available,  19.625 MB used (8 %)
    VM Heap  memory : 159.996 MB available,   0.846 MB used (0 %)

    nil
    : (VLE-FILE->LIST "d:\\Downloads\\GD201113-087-R\\30mb.xyz" NIL)
    ("X Y Z" "2716000.......)

    : (mem)

    GC calls Total  256 [3.03 secs.]  (256 in all docs)
    GC calls System 0, Int 0, Real 0, Vector 0, Strg 3, Symbol 0, Cons 0, Heap 253

    VM total memory : 399.996 MB
    VM Page  memory : 200.000 MB available,  53.875 MB used (26 %)
    VM Heap  memory : 199.996 MB available, 135.870 MB used (67 %)

    nil
    : (gc)
    nil
    : (mem)

    GC calls Total  257 [3.05 secs.]  (257 in all docs)
    GC calls System 1, Int 0, Real 0, Vector 0, Strg 3, Symbol 0, Cons 0, Heap 253

    VM total memory : 399.996 MB
    VM Page  memory : 200.000 MB available,  53.875 MB used (26 %)
    VM Heap  memory : 199.996 MB available,   0.448 MB used (0 %)

    nil
    : (VLE-FILE->LIST "d:\\Downloads\\GD201113-087-R\\50mb.xyz" NIL)

    ; ----- Error around expression -----
    ; (#<<external-object> fffffffffffffff4> #<<external-object> fffffffffff

Torsten Moses

Dear James,

Is that 250 mb memory limit similar or different from acad?

the Lisp engine reserves those 250 MB at startup (not really allocated from process memory, just reserved only - to have an ambiguous memory block); that reserved memory block is splitted into "page" and "heap", and both ranges can dynamically grow or shrink, if the other range gets full ... until (nearly) all of that memory block is used.

As fallback, there is the option to increate that memory, via .cfg file ...

AutoCAD AutoLISp most likely uses memory the "normal way", dynamically allocating when more memory is needed ...
but this obviously causes significant management overhead - that different memory handling is one of the main reasons for our better LISP performance (but not the only one) ... drawback is in such "exotic" cases as the monster lists here :-)

Dear Peter,

yes, likely the smarter approach, to process "per file line", for such big data ... or at least, to read in chunks, say 1000 points per read operations (using a small helper Lisp function).

many greetings !

James Maeding

@Torsten Moses
Interesting. I guess I can say I have no complaints yet with bcad lisp.
On the lisp speed radar, I would say the number one issue I have seen is people using scripting like "(command....)" instead of entmod or entmake.
They are driving the car in 1st gear and that poor engine is dying to get to 3rd gear.

Roy Klein Gebbinck

If you are really interested in speed you should switch to using vle-entget/vle-entmod and the vl* (Fast-COM!) functions in BricsCAD.

Torsten Moses

Dear James,

yes, the (command ....) is the slowest part, indeed
AutoCAD does some "dirty" tricks to stop display updates with (command), up to a certain amount ... partially we do similar tricks, but not that sophisticated as AutoLISP does.

Besides, in our Lisp we also have (vle-start/end-transaction), also (vle-displaypause) - also emulated as NoOperation for AutoCAD - this can help to further improve performance of Lisp code ...

many greetings !

Peter2

@Torsten Moses

actually, our LISP engine uses a "fixed" memory range (usually ~200 MB) ... so indeed, huge lists can cause out-of-memory conditions ...
3 solutions :
a) there is lispex.dll.cfg where you can define memory limits ....

I want to reanimate this topic with a short question:
Will there be a better handling of this setting in the future?
As mentioned above, the file is one setting on one PC under admin-rights. A setting at runtime - like setvar or get_ini or read_registry - would be "state of the art"

Torsten Moses

Dear Peter,

actual V21.1.06 has fixed the switched options, so those do as they are documented for.
The memory presets/limits are used at Lisp engine initialisation ... they can not be changed at runtime, would have no effects :-(
But I could add a runtime function to set those 3 values via a Lisp function, then store to registry - and could have priority over the .cfg file, at next startup ...
Would that be as you imagine ?
many greetings !

James Maeding

in HKEY_CURRENTUSER?
would not want that setting in admin only spot...

Torsten Moses

in HKEY_CURRENT_USER
that is normal user's part, no admin access required ...

Peter2

Maybe to be configured and stored via "Profile"? I think it should be visible to the users, but for the technical solution I trutst on your experience.

Peter2

@Torsten Moses

@Torsten Moses said:
...
But I could add a runtime function to set those 3 values via a Lisp function, then store to registry - and could have priority over the .cfg file, at next startup ...

I'm preparing to deploy V21.2.02 in my company. What shall I do to have the correct setting on every machine during installation/configuration?

Torsten Moses

Dear Peter :-)

still the only option, to distribute a specific lispex.dll.cfg into each installation folder;
the parameters do work as documented, the defect with switched option is fixed in V21.2.02.

So that should work then for you.
many greetings !