I just finished programming the hardest thing I ever programmed in my life. The goal of my recent research was how to implement a scalable mmorpg server that could support hundreds of players at one time. I immediately agreed that this kind of server would have to be multi threaded, little did I know that the whole thing would end up running on over 15 threads. The majority of the work is done with Newton and Raknet as they ended up taking 13 of those threads. Now RakNet was never a problem, it is a wonderful library that hasnt ever crashed for me for whatever reason. The problem was Newton Dynamics, not in the library itself (Its a great library) but the fact that it was pretty hard to avoid racing conditions and heap corruptions due to the fact that I was running Newton on its own thread plus spawning 8 internal Newton threads.
After a pretty long time debugging and tuning the code I managed to get it to run without any heap corruptions/racing conditions using batching. I had to split the character controllers with the player objects in order to do that. I was using some STL algorithms to push players who needed a controller made into a vector, then sending that vector as a part of a batch to the collision thread to process, while it was processing I had to Sleep(1) on main thread. The whole system ended up being very slow because of unnecessary re allocations of the vector.
Then as I was speaking to my friend on MSN I had a brilliant idea, why waste all this processing power on making these damn controllers when I could just make a static sized array of them right at the beginning? It was a brilliant idea because the actual controllers themselves didn't take up that much memory.
That worked brilliantly except for the fact that now Newton was iterating through a large amount of bodies and doing needless processing. Unfortunately there isn't a way to NOT add a body to the Newton world when making it, but there is a way to make it freeze and not be processed (Although still collidable with other controllers).
So I ended up assigning raw controller pointers to players who joined and incrementing the controller index
so the next player can get the next controller along. Also freezing and unfreezing controllers as players join/leave. (Controllers get made frozen by default).
The last problem was to prevent the collision between frozen controllers and unfrozen controllers. Well that simple, just change the material id's and material contact callbacks to appropriate bodies. just make sure the contact callback returns 0 and the second one simply returns doing nothing.
This whole architecture turned out to resemble a library, you can borrow a book (Character controller) from the bookshelf (Array of controllers) and return it it in the condition that you received it (Reset controller position to default and any states that it had prior to when you borrowed it).
Some of my server features now
Dynamic threading
Depending on the hardware it can support up to 1000 players per zone with unlimited zone capabilities
Lockless design
Scalable solver models and architectures thanks to newton
Great cohesion between controllers and players
Still stuff to be implemented:
Hash table/Voronoi diagram? for player proximity calculations
Load balancing
