239 - Grail.quests takes too much space

Grail.quests takes about 4m Lua memory, which is pretty big by WoW standards. It wastes lots of memory on overhead because every quest is a small two-element table. This also negatively affects GC speed. Converting thousands of those small tables to just two big tables will save about 15%, give small speed boost to GC and most code can be adapted to it with little manual work - most changes are pretty much search-and-replace.

Grail.quest_names = {}
Grail.quest_flags = {}

for key, val in pairs(Grail.quests do
   Grail.quest_names[key] = val[1]
   Grail.quest_flags[key] = val[2]
Grail.quests = nil

(BTW, NPCs can be processed in that way too. I'm not quite sure why you duplicate NPC ID as first element).

User When Change
Nimhfree Dec 13, 2013 at 18:41 UTC Changed status from Accepted to Fixed
Nimhfree Mar 23, 2013 at 18:04 UTC Changed status from New to Accepted
rowaasr13 Mar 22, 2013 at 11:26 UTC Create

You must login to post a comment. Don't have an account? Register to get one!

  • Avatar of Nimhfree Nimhfree Dec 13, 2013 at 18:40 UTC - 0 likes

    Grail 059 has some more optimizations. Over time more will be made as appropriate so I think this can be closed.

  • Avatar of Nimhfree Nimhfree May 11, 2013 at 19:29 UTC - 0 likes

    Another experiment. With the release of Grail 048 the prerequisites are not being processed into a table structure. Instead they are remaining as a string and processing is done each time the information needs to be handled. This saves about 1.0 MB of space.

  • Avatar of Nimhfree Nimhfree May 07, 2013 at 21:20 UTC - 0 likes

    I have been experimenting a little. I have changed the way Grail-Achievements and Grail-Reputations work so they transform the data that are in the little tables into other structures. Doing so reduces each of their footprints by 0.6MB. I have also taken the four integers from the Grail.quests table (indexes 2, 3, 4, 13) and transformed them into one string instead. This saves a little space, but nothing like removing little tables. I had attempted to do the same with the computed status (index 7), but failed to get that done properly, so the code is commented out for the release.

    I believe the biggest gain will be to transform the structures that are used to represent things like prerequisites into a format that does not use tables. Perhaps keeping them strings and getting better evaluation routines to minimize table creation but maintains speed will be best.

  • Avatar of Nimhfree Nimhfree Mar 24, 2013 at 19:10 UTC - 0 likes

    On the official prerelease page I have posted a version of Grail that has these changes to the NPC structure present.

  • Avatar of Nimhfree Nimhfree Mar 23, 2013 at 19:08 UTC - 0 likes

    I did an experiment with the Grail.npcs table. At the end of the Grail-NPCs.lua file I put this code:

    --		[1] maps into Grail.npcIndex
    --		[2] maps into Grail.npcCodes
    --		[3] maps into Grail.npcComments
    --		[4] maps into Grail.npcFactions
    Grail.npcIndex = {}
    Grail.npcCodes = {}
    Grail.npcComments = {}
    Grail.npcFactions = {}
    for key, value in pairs(Grail.npcs) do
    	Grail.npcIndex[key] = value[1]
    	if value[2] then Grail.npcCodes[key] = value[2] end
    	if value[3] then Grail.npcComments[key] = value[3] end
    	if value[4] then Grail.npcFactions[key] = value[4] end
    Grail.npcs = nil
    print("Done converting NPCs with elapsed milliseconds: "..debugprofilestop())

    On one of my test machines the elapsed time is about 10 to 11.

    Before applying these changes, I looked at the memory for Grail in general after sitting idle for a couple minutes after loading. ACP reports 18.84 Mb, and Blizzard reports 19.29 Mb.

    After applying these changes (and of course changing the code in Grail.lua to use these new tables) the same memory reports as 18.25 Mb and 18.69 Mb respectively, showing a 0.6 Mb decrease in memory size.

  • Avatar of Nimhfree Nimhfree Mar 23, 2013 at 14:03 UTC - 0 likes

    The flag data (Grail.quests[questId][1]) can be erased after it is parsed.

    I will be glad to find from you whether the thousands of small tables are worse than the few large tables you propose. It is good to have you able to do this while I concentrate on other aspects. Thanks again.

  • Avatar of rowaasr13 rowaasr13 Mar 23, 2013 at 00:58 UTC - 0 likes

    Yeah, after diving a bit deeper I noticed that you dump parsed data to same table. This makes my task somewhat longer, but I'll manage. :) I expect this will also make code a little more readable after, because named tables are better than cryptic numeric indices.

    BTW, do you need serialized flag data after you've parsed it? If not, it can be safely dropped for some extra memory savings after _CodeAllFixed had its way with the string.

  • Avatar of Nimhfree Nimhfree Mar 22, 2013 at 21:22 UTC - 0 likes

    Yes, that entry in Grail.npcs indicates that NPC 248 has its name as Grail.npcNames[248]. If you look at Grail.npcs[61693]={59354,'809[9]:64.64,60.92'}, for example, this means the name is really Grail.npcNames[59354] which is 'Muskpaw Jr.'. Most of the NPCs will probably have the same first element as their index. But there are lots that do not. It actually seems to have saved memory from the basic testing I performed.

    By the way, the Grail.quests initially has only two members when the Lua file is read in. However, after processing the number of members increases a bunch. From the comments in Grail-Quests.lua:

    0 localized name of the quest

    1 code string

    2 cached type bit value

    3 cached holiday bit value

    4 cached obtainers bit value

    5 achievement "map areas"

    6 table of reputation changes when quest turned in

    7 realtime computed status

    8 A codes (NPC IDs processed for faction)

    9 T codes (NPC IDs processed for faction)

    10 P codes

    11 I codes

    12 O codes

    13 cached level bit value

    14 reputation requirements from P codes (a table of tables where the internal table has the reputation code and the amount (with amount negative meaning not to exceed))

    Note that there are still character values stored as well since not everything has been converted to the integer values yet (and not all of these are used). If you really need to see during play just dump out the table for any Grail.quests[xxx] that exists to see what has been cached, etc. Note that after the addon is loaded Grail.quests[xxx][1] can be set to nil if really needed as all the information should have been taken from it and used to fill lots of other tables. I am not sure how your testing of converting the table structure will be impacted based on this information.

    Last edited Mar 22, 2013 by Nimhfree
  • Avatar of rowaasr13 rowaasr13 Mar 22, 2013 at 12:32 UTC - 0 likes

    No, this runtime conversion is just an example. It is much better to generate it that way from the start, of course.

    I'm talking about Grail.npcs table. Typical entry I see right now is:
     ^^^^    ^^^^

    Last edited Mar 22, 2013 by rowaasr13
  • Avatar of Nimhfree Nimhfree Mar 22, 2013 at 12:05 UTC - 0 likes

    Are you proposing the data structure stay the way it is in the source files, but have it massaged into the new table structure at runtime?

    The reason the NPCs are set up the way they are is because that first element is an index in the NPC names. Basically the NPC names are uniqued for English values. All NPCs with the same name should have that single first element. This also means that localized names that happen to match the English values will not need to be localized. With the way things work, there is not uniquing being done between non-English names, so if the Italian and Spanish versions of an NPC's name are the same, but that value is different from English, there will be two entries into the localized values (because those localized values are only processed for a single language, overwriting the English values).

    I welcome your work. I will try to get the latest data up on the prerelease page either today or tomorrow. There is some work being done on getting phasing implemented as prerequisites for NPCs, but that will not be done soon. It should not really impact the quest name/code sections of code so should minimally impact your code. I assume you have the ability to diff the different versions to see what changes were made to ease your work.



Last updated
Dec 13, 2013
Mar 22, 2013
Fixed - Developer made requested changes. QA should verify.
Enhancement - A change which is intended to better the project in some way
Medium - Normal priority.

Reported by

Possible assignees