XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Sun Oct 22, 2017 2:57 pm

All times are UTC + 1 hour




Post new topic Reply to topic  [ 4 posts ] 
Author Message
 Post subject: How to speed up your QuickBMS scripts
PostPosted: Tue Sep 04, 2012 1:31 am 
Offline
double-veteran
double-veteran

Joined: Mon Aug 24, 2009 10:55 pm
Posts: 984
Has thanked: 76 times
Have thanks: 596 times

When processing large files or small portions of data in huge loops, scripts can be really slow. Often it's because of faulty code. So here are some general tips for QuickBMS coders to make your scripts faster. :) This list may be updated in the future.

1. Avoid double calculation
avoid calculating variables inside a loop when you can do so outside. Example:
Code:
set HEADER 0x800
set DAT 0x100
for i = 1 <= 10000
    get FSIZE asize      # same operation each loop
    math FSIZE -= HEADER #              "
    set SUB i
    math SUB *= DAT
    math FSIZE -= SUB
next i

should be corrected to
Code:
set HEADER 0x800
set DAT 0x100
get TSIZE asize
math TSIZE -= HEADER # one calculation before the loop
for i = 1 <= 10000
    set FSIZE TSIZE # just take the already calculated variable
    set SUB i
    math SUB *= DAT
    math FSIZE -= SUB
next i



2. Avoid appending (thanks for this tip, Luigi :) )
Instead of
Code:
get CYCLES asize
math CYCLES /= 8
set OFFSET 0
append
for i = 1 <= CYCLES
    log MEMORY_FILE OFFSET 8
    math OFFSET += 8
next i
append

use
Code:
get CYCLES asize
math CYCLES /= 8
for i = 1 <= CYCLES
    getDstring DATA 8
    putDstring DATA 8 MEMORY_FILE
next i



3. Pre-allocate memory
Whenever you have to write something to MEMORY_FILE, set it's size first:
Code:
putVarChr MEMORY_FILE PRE_SIZE 0
log MEMORY_FILE 0 0

If you don't know the size of the complete file without running through the data and getting single block sizes (e.g. when demultiplexing certain video formats), it's faster to pre-allocate the memory in one loop and then fill it in a second one:
Code:
set PSIZE 0
savepos MYOFF
do
    get MSIZE long
    savepos OFFSET
    math PSIZE += MSIZE
    putVarChr MEMORY_FILE PSIZE 0
    log MEMORY_FILE 0 0
while [ending argument]
goto MYOFF
do
    get MSIZE long
    savepos TEMPOFF
    getDstring DATA MSIZE
    putDstring DATA MSIZE MEMORY_FILE
    math TEMPOFF += MSIZE
    goto TEMPOFF
while [ending argument]


You can make the ads go away by registering


_________________
If you like what you see, why not click the little Thank You button? ;) It will definitely motivate me! :)
And here's Mr.Mouse's Facebook link: http://www.facebook.com/permalink.php?s ... 8469022795 - thanks ;)


Last edited by AlphaTwentyThree on Tue Sep 04, 2012 1:15 pm, edited 3 times in total.

Top
 Profile  
 
 Post subject: Re: How to speed up your QuickBMS scripts
PostPosted: Tue Sep 04, 2012 2:08 am 
Offline
Site Admin
User avatar

Joined: Mon Sep 13, 2004 1:55 am
Posts: 703
Location: The wiki.
Has thanked: 107 times
Have thanks: 122 times
A lot of this stuff is really applicable to programming in general: whenever you have a loop, you want it as "tight" as possible (meaning it does as little as possible), to allow the program to get through it faster. Actually, on that note, I wonder if your second "Pre-allocate memory" example would be even faster if you separated the 2-cycle loop into separate loops, so there's no overhead from the If block:

Code:
set PSIZE 0
do
    get MSIZE long
    savepos OFFSET
    math PSIZE += MSIZE
while [ending argument]
putVarChr MEMORY_FILE PSIZE 0
log MEMORY_FILE 0 0
do
    get MSIZE long
    savepos OFFSET
    append
    log MEMORY_FILE OFFSET MSIZE
    append
while [ending argument]

(also, is there a way to lose the append pair in the second loop? using it goes against your own advice to avoid it, after all ;) )

AlphaTwentyThree wrote:
Code:
]get CYCLES asize
math CYCLES /= 8
for i = 1 <= CYCLES
    getDstring DATA 8
    putDstring DATA 8 MEMORY_FILE
next i

Is the extra "]" meant to be at the beginning of that?

_________________
Welcome to Xentax!

Rules | Requests | Wiki

If you run across a post that breaks the rules, please PM Craptain, Mr.Mouse, or myself - we will handle it from there.


Top
 Profile  
 
 Post subject: Re: How to speed up your QuickBMS scripts
PostPosted: Tue Sep 04, 2012 4:38 am 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran

Joined: Sat Apr 09, 2011 1:22 am
Posts: 2387
Has thanked: 170 times
Have thanks: 277 times
Is the reason for dropping the append for purely I/O reasons? Or is there something else to it?

_________________
Model Import Plugins/Scripts

Noesis Plugins | Reference files

Reference
Guide to 3D model reversal | 3D glossary


Top
 Profile  
 
 Post subject: Re: How to speed up your QuickBMS scripts
PostPosted: Tue Sep 04, 2012 1:11 pm 
Offline
double-veteran
double-veteran

Joined: Mon Aug 24, 2009 10:55 pm
Posts: 984
Has thanked: 76 times
Have thanks: 596 times
Dinoguy1000 wrote:
Actually, on that note, I wonder if your second "Pre-allocate memory" example would be even faster if you separated the 2-cycle loop into separate loops, so there's no overhead from the If block

Yeah, you're right. ;) Changed it. :)

Dinoguy1000 wrote:
Also, is there a way to lose the append pair in the second loop? using it goes against your own advice to avoid it, after all ;)

Hm, I haven't tested that. I've changed it BUT it could be slower because the variable (i.e. the data for the memory file) is really big and there's no pre-allocated memory for it I think. If I'm wrong, please correct me Luigi. ;) If so, it would of course be better to add another if-case to use getDstring for MSIZE < [some value] and the append and log command for data that's bigger than this value. Would be nice to know, so if you see this, Luigi, please post!

Dinoguy1000 wrote:
Is the extra "]" meant to be at the beginning of that?

Nope. ;) Corrected.

finale00 wrote:
Is the reason for dropping the append for purely I/O reasons? Or is there something else to it?

No, just that the speed gain is immense. ;)


_________________
If you like what you see, why not click the little Thank You button? ;) It will definitely motivate me! :)
And here's Mr.Mouse's Facebook link: http://www.facebook.com/permalink.php?s ... 8469022795 - thanks ;)


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group