Join also our Discord channel! Click here.

How to speed up your QuickBMS scripts

Coders and would-be coders alike, this is the place to talk about programming.
Post Reply
AlphaTwentyThree
double-veteran
double-veteran
Posts: 984
Joined: Mon Aug 24, 2009 10:55 pm
Has thanked: 75 times
Been thanked: 634 times

How to speed up your QuickBMS scripts

Post by AlphaTwentyThree » Tue Sep 04, 2012 1:31 am

When processing large files or small portions of data in huge loops, scripts can be really slow. Often it's because of faulty code. So here are some general tips for QuickBMS coders to make your scripts faster. :) This list may be updated in the future.

1. Avoid double calculation
avoid calculating variables inside a loop when you can do so outside. Example:

Code: Select all

set HEADER 0x800
set DAT 0x100
for i = 1 <= 10000
    get FSIZE asize      # same operation each loop
    math FSIZE -= HEADER #              "
    set SUB i
    math SUB *= DAT
    math FSIZE -= SUB
next i
should be corrected to

Code: Select all

set HEADER 0x800
set DAT 0x100
get TSIZE asize
math TSIZE -= HEADER # one calculation before the loop
for i = 1 <= 10000
    set FSIZE TSIZE # just take the already calculated variable
    set SUB i
    math SUB *= DAT
    math FSIZE -= SUB
next i

2. Avoid appending (thanks for this tip, Luigi :) )
Instead of

Code: Select all

get CYCLES asize
math CYCLES /= 8
set OFFSET 0
append
for i = 1 <= CYCLES
    log MEMORY_FILE OFFSET 8
    math OFFSET += 8
next i
append
use

Code: Select all

get CYCLES asize
math CYCLES /= 8
for i = 1 <= CYCLES
    getDstring DATA 8
    putDstring DATA 8 MEMORY_FILE
next i

3. Pre-allocate memory
Whenever you have to write something to MEMORY_FILE, set it's size first:

Code: Select all

putVarChr MEMORY_FILE PRE_SIZE 0
log MEMORY_FILE 0 0
If you don't know the size of the complete file without running through the data and getting single block sizes (e.g. when demultiplexing certain video formats), it's faster to pre-allocate the memory in one loop and then fill it in a second one:

Code: Select all

set PSIZE 0
savepos MYOFF
do
    get MSIZE long
    savepos OFFSET
    math PSIZE += MSIZE
    putVarChr MEMORY_FILE PSIZE 0
    log MEMORY_FILE 0 0
while [ending argument]
goto MYOFF
do
    get MSIZE long
    savepos TEMPOFF
    getDstring DATA MSIZE
    putDstring DATA MSIZE MEMORY_FILE
    math TEMPOFF += MSIZE
    goto TEMPOFF
while [ending argument]
Last edited by AlphaTwentyThree on Tue Sep 04, 2012 1:15 pm, edited 3 times in total.
If you like what you see, why not click the little Thank You button? ;) It will definitely motivate me! :)
And here's Mr.Mouse's Facebook link: http://www.facebook.com/permalink.php?s ... 8469022795 - thanks ;)

User avatar
Dinoguy1000
Site Admin
Posts: 743
Joined: Mon Sep 13, 2004 1:55 am
Has thanked: 113 times
Been thanked: 136 times

Re: How to speed up your QuickBMS scripts

Post by Dinoguy1000 » Tue Sep 04, 2012 2:08 am

A lot of this stuff is really applicable to programming in general: whenever you have a loop, you want it as "tight" as possible (meaning it does as little as possible), to allow the program to get through it faster. Actually, on that note, I wonder if your second "Pre-allocate memory" example would be even faster if you separated the 2-cycle loop into separate loops, so there's no overhead from the If block:

Code: Select all

set PSIZE 0
do
    get MSIZE long
    savepos OFFSET
    math PSIZE += MSIZE
while [ending argument]
putVarChr MEMORY_FILE PSIZE 0
log MEMORY_FILE 0 0
do
    get MSIZE long
    savepos OFFSET
    append
    log MEMORY_FILE OFFSET MSIZE
    append
while [ending argument]
(also, is there a way to lose the append pair in the second loop? using it goes against your own advice to avoid it, after all ;) )
AlphaTwentyThree wrote:

Code: Select all

]get CYCLES asize
math CYCLES /= 8
for i = 1 <= CYCLES
    getDstring DATA 8
    putDstring DATA 8 MEMORY_FILE
next i
Is the extra "]" meant to be at the beginning of that?
Welcome to Xentax!

Rules | Requests | Wiki | Discord

If you run across a post that breaks the rules, please report the post - a mod or admin will handle it from there.

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 297 times

Re: How to speed up your QuickBMS scripts

Post by finale00 » Tue Sep 04, 2012 4:38 am

Is the reason for dropping the append for purely I/O reasons? Or is there something else to it?

AlphaTwentyThree
double-veteran
double-veteran
Posts: 984
Joined: Mon Aug 24, 2009 10:55 pm
Has thanked: 75 times
Been thanked: 634 times

Re: How to speed up your QuickBMS scripts

Post by AlphaTwentyThree » Tue Sep 04, 2012 1:11 pm

Dinoguy1000 wrote:Actually, on that note, I wonder if your second "Pre-allocate memory" example would be even faster if you separated the 2-cycle loop into separate loops, so there's no overhead from the If block
Yeah, you're right. ;) Changed it. :)
Dinoguy1000 wrote:Also, is there a way to lose the append pair in the second loop? using it goes against your own advice to avoid it, after all ;)
Hm, I haven't tested that. I've changed it BUT it could be slower because the variable (i.e. the data for the memory file) is really big and there's no pre-allocated memory for it I think. If I'm wrong, please correct me Luigi. ;) If so, it would of course be better to add another if-case to use getDstring for MSIZE < [some value] and the append and log command for data that's bigger than this value. Would be nice to know, so if you see this, Luigi, please post!
Dinoguy1000 wrote:Is the extra "]" meant to be at the beginning of that?
Nope. ;) Corrected.
finale00 wrote:Is the reason for dropping the append for purely I/O reasons? Or is there something else to it?
No, just that the speed gain is immense. ;)
If you like what you see, why not click the little Thank You button? ;) It will definitely motivate me! :)
And here's Mr.Mouse's Facebook link: http://www.facebook.com/permalink.php?s ... 8469022795 - thanks ;)

Post Reply