XeNTaX

Posted: **Tue Sep 04, 2012 1:31 am**

When processing large files or small portions of data in huge loops, scripts can be really slow. Often it's because of faulty code. So here are some general tips for QuickBMS coders to make your scripts faster.

This list may be updated in the future.

1. Avoid double calculation
avoid calculating variables inside a loop when you can do so outside. Example:

Code: Select all

set HEADER 0x800
set DAT 0x100
for i = 1 <= 10000
    get FSIZE asize      # same operation each loop
    math FSIZE -= HEADER #              "
    set SUB i
    math SUB *= DAT
    math FSIZE -= SUB
next i

should be corrected to

Code: Select all

set HEADER 0x800
set DAT 0x100
get TSIZE asize
math TSIZE -= HEADER # one calculation before the loop
for i = 1 <= 10000
    set FSIZE TSIZE # just take the already calculated variable
    set SUB i
    math SUB *= DAT
    math FSIZE -= SUB
next i

2. Avoid appending (thanks for this tip, Luigi

)
Instead of

Code: Select all

get CYCLES asize
math CYCLES /= 8
set OFFSET 0
append
for i = 1 <= CYCLES
    log MEMORY_FILE OFFSET 8
    math OFFSET += 8
next i
append

use

Code: Select all

get CYCLES asize
math CYCLES /= 8
for i = 1 <= CYCLES
    getDstring DATA 8
    putDstring DATA 8 MEMORY_FILE
next i

3. Pre-allocate memory
Whenever you have to write something to MEMORY_FILE, set it's size first:

Code: Select all

putVarChr MEMORY_FILE PRE_SIZE 0
log MEMORY_FILE 0 0

If you don't know the size of the complete file without running through the data and getting single block sizes (e.g. when demultiplexing certain video formats), it's faster to pre-allocate the memory in one loop and then fill it in a second one:

Code: Select all

set PSIZE 0
savepos MYOFF
do
    get MSIZE long
    savepos OFFSET
    math PSIZE += MSIZE
    putVarChr MEMORY_FILE PSIZE 0
    log MEMORY_FILE 0 0
while [ending argument]
goto MYOFF
do
    get MSIZE long
    savepos TEMPOFF
    getDstring DATA MSIZE
    putDstring DATA MSIZE MEMORY_FILE
    math TEMPOFF += MSIZE
    goto TEMPOFF
while [ending argument]

Posted: **Tue Sep 04, 2012 2:08 am**

A lot of this stuff is really applicable to programming in general: whenever you have a loop, you want it as "tight" as possible (meaning it does as little as possible), to allow the program to get through it faster. Actually, on that note, I wonder if your second "Pre-allocate memory" example would be even faster if you separated the 2-cycle loop into separate loops, so there's no overhead from the If block:

Code: Select all

set PSIZE 0
do
    get MSIZE long
    savepos OFFSET
    math PSIZE += MSIZE
while [ending argument]
putVarChr MEMORY_FILE PSIZE 0
log MEMORY_FILE 0 0
do
    get MSIZE long
    savepos OFFSET
    append
    log MEMORY_FILE OFFSET MSIZE
    append
while [ending argument]

(also, is there a way to lose the append pair in the second loop? using it goes against your own advice to avoid it, after all

)

AlphaTwentyThree wrote:

Code: Select all

]get CYCLES asize
math CYCLES /= 8
for i = 1 <= CYCLES
    getDstring DATA 8
    putDstring DATA 8 MEMORY_FILE
next i

Is the extra "]" meant to be at the beginning of that?

Posted: **Tue Sep 04, 2012 4:38 am**

Is the reason for dropping the append for purely I/O reasons? Or is there something else to it?

Posted: **Tue Sep 04, 2012 1:11 pm**

Dinoguy1000 wrote:Actually, on that note, I wonder if your second "Pre-allocate memory" example would be even faster if you separated the 2-cycle loop into separate loops, so there's no overhead from the If block

Yeah, you're right.

Changed it.

Dinoguy1000 wrote:Also, is there a way to lose the append pair in the second loop? using it goes against your own advice to avoid it, after all

Hm, I haven't tested that. I've changed it BUT it could be slower because the variable (i.e. the data for the memory file) is really big and there's no pre-allocated memory for it I think. If I'm wrong, please correct me Luigi.

If so, it would of course be better to add another if-case to use getDstring for MSIZE < [some value] and the append and log command for data that's bigger than this value. Would be nice to know, so if you see this, Luigi, please post!

Dinoguy1000 wrote:Is the extra "]" meant to be at the beginning of that?

Nope.

Corrected.

finale00 wrote:Is the reason for dropping the append for purely I/O reasons? Or is there something else to it?

No, just that the speed gain is immense.

XeNTaX

How to speed up your QuickBMS scripts

How to speed up your QuickBMS scripts

Re: How to speed up your QuickBMS scripts

Re: How to speed up your QuickBMS scripts

Re: How to speed up your QuickBMS scripts