We fulfill power fantasies

RSS Feed

Give me proper networking tools on Windows

10.11.2018 21:44:09

Last week I spent multiple days creating a bad, inefficient abstraction over WSAPoll, which itself is a bad, inefficient abstraction of other functions, simply provided for UNIX compatibility by Microsoft.

The thing is, Windows has efficient, scalable methods of managing network IO, namely, IO completion ports. But it's such a pain to use in a cross-platform application. With IOCP, because you only get informed after a recv() on a socket is done, not when you can do one yourself, you must maintain unique buffers on each one of your sockets. And if your application is a cross-platform one, you basically have to make your Linux abstraction work in a similar fashion, because the socket recv() function just cannot be used the same way with both systems. (As a side note, I've built an API for async socket IO that works on both systems, using IOCP on Windows and epoll on Linux. It's called netqueue and it's included in MUTA's source code.

Linux on the other has epoll, one of the comfiest APIs I've every used. I love epoll in terms of how easy to use it is. Add a file descriptor or many to an epoll instance and just sleep on epoll_wait() until something happens, like you receive some data over the network.

I want epoll on Windows! But I can't get it, not without additional dependencies to external libraries or a lot of work.

My saving grace is that our game's servers don't need to run on Windows in release mode. The only reason the server runs on Windows is that sometimes we have to develop on it, and in such cases its handy to have the server run on the same machine with the client. So what I've done is I've wrapped WSAPoll to look a bit like epoll. It's a terrible hack and certainly inefficient, but it'll do for our purposes. Of course, features such as EPOLLET, edge-triggered behaviour, will not work on Windows (in edge-triggered mode, new events for sockets are only notified of once, not until the socket is recv()d from), and in fact even a library I found, attempting to provide epoll for Windows, didn't support said feature.

Still. I want epoll or something similar to it on Windows. Why does Win32 programming have to always be such a pain?

MUTA devlog 4: proxy and login servers and future art direction

2.11.2018 17:48:17

The past few months have granted me little time to work on MUTA, our 2D MMORPG. There have been events to attend (NGS), another trip to India (IGS), and work. But I've done a bit of MMO work nevertheless.

The proxy server

I began working on MUTA's proxy server. It is a server that sits between the client and the game server, dispatching messages in both directions. The purpose of it is to never reveal the true IP address of the game server, making it more resilient towards DDOS attacks. Each proxy could be connected to multiple game servers, and each game server to multiple proxies, so that two people playing on the same server might be connected to completely different IP addresses.

I'm writing the initial implementation using TCP, in C as the rest of the game is written in. Adding another layer of connectivity, especially with TCP, will of course increase latency, so it remains to be seen if we should do the game-server-to-proxy connection via UDP later.

The idea for the proxy came from many sources, but one of them is CipSoft's Matthias Rudy's GDC talk about the infrastrucuture of Tibia. The slides for the talk are here.

The login server

Once I had sat down and begun writing the proxy server, I went as far as accepting clients and reading their packets before I realized I should probably write the login server at this point, too. Currently in MUTA, the game server acts as a temporary login server. It wouldn't make much sense to write login-related functionality code between the proxy and game servers, only to remove it later. So I also started up the login server project.

Event based architecture

On the login server I thought I might try out something that's new for me. I'm not a very experienced network programmer, and hence I haven't had the chance to try out that many different tricks yet.

Dividing work between threads is still an issue I constantly run into when writing server software. Usually, because of my background in game programming, I've simply settled for having a main loop that runs at a certain frame rate, and the other threads feed work to that thread, such as handling incoming accepts or reads. Work is fed using various buffers that store different kinds of events (for example, a client received something in it's buffer, the message should be handled). This is not very elegant or energy-efficent design.

On the login server, I thought I might still have one thread executing the main logic, but instead of locking the thread's frame rate, I'll try having the main thread sleep until it gets work from other threads. I will also try to unify the event buffers so that most events would be inserted into a single queue. Insertion would block the feeding thread if the queue is full. Well, we'll see how that goes.

Art direction

Something thing we've recently (and before, too) been pondering with Lommi is what would be the most practical direction we could take in regards to art in MUTA. It's been a tough subject!

We know the feeling we want the game to go for: the sort of gritty fantasy provided by Robert E. Howard's Conan stories and other sword-and-sorcery works. We also really dig the artstyle of many of the comics made of Conan. In a 2D game that would look awesome. But 2D art in 8 isometric directions is painstaking, and it's complicated by things such as displaying equipment on characters while keeping animations smooth-looking.

We thought of a couple of directions we could follow. One would be to reduce the game's resolution from 64 pixel tiles to 32 or 16 pixels. Such a low resolution would allow even us programmers to make art for the game, albeit low quality art. We didn't really like the idea, as it doesn't let us present the game world as such a gritty, "mature" type of place as we'd like, and it would limit the art style a lot. Besides, the programmers probably shouldn't do the art anyway.

Another thought we had was going 3D. Neither of us two working on the game can do 3D art, but 3D would mean not needing to draw 8 different directions for each object (or at least each character). Then again, 3D would probably present our style in quite a bit of a worse manner than well-made 2D in isometric space.

We didn't decide anything yet, other than that the direction we have sought so far would look the best.

Going to India Game Summit 2018

21.9.2018 21:46:40

As I wrote last month, I travelled to Andhra Pradesh, India during the summer to teach game programming on a summer course over at a university there. At the end of this September (less than a week from now) I have the pleasure of going back to the country, this time as an organizer for the first ever India Game Summit.

The event this year will be a game jam, held at four different universities and colleges across the country simultaneously. How ever, it is in the hopes of organizers (the main one of whom is Kajaani University of Applied Sciences) that the event will grow into a games industry conference in the future, something akin to the Northern Game Summit we have in Finland, with speakers from different companies and organizations sharing their knowledge.

An interesting point about organizing a game jam event in India is that it seems to be a fairly foreign concept, at least at universities (including the capital area). Even the term game jam has been mostly unheard of; hence, we've dubbed the event a hackathon instead - a term familiar to most IT people. Organizers have also been contacted by to-be participants worried if it is even possible to build a game in 2 days. It may seem funny to a regular game jammer, but then again there are monetary prizes on the line here.

The event will begin on 28th September and end on the 30th of the same month. I myself will be travelling to Aditya Engineering College in Andhra Pradesh. I'm quite excited and interested to see how things work out, to be honest.

New website logo

In other news, this website has a fancy new logo. Our friend Laura made it. Big thanks to her!

A Game About Carrots

5.8.2018 16:02:35

I've had little time for hobby programming in the past two months due to working abroad. However, while in India my colleague, Laura (who was also there) told me she wanted to make a game. She began drawing some pixel art and on the off hours, I wrote some good old C code.

The game we began writing is Porkkana, Finnish for the word carrot. The game concept is very small and, uh, sort of unclear. Actually, I'm not sure we completely even know what the game is going to be about yet, other than that you grow carrots in it. And that you feed the carrots to bunnies. There's probably going to be some sort of trouble coming your way though, like carrots dying or getting stolen. Who knows.

Most of the engine code I was able to steal from MUTA, my primary project at this time. I copy pasted the OpenGL rendering and the immediate mode GUI system as well as various utilities I always use from different projects. The gameplay code itself is in one file at this point and less than 1000 lines long. The amount of hours spent on programming the game is actually very low, but every now and then I might spend an hour or two on the code in the evening. The most time-consuming thing so far has been getting Android building to work; it does now, but getting to that point took probably at least a full 8 hour work day's worth of time. If there's one thing I hate in this world, it's the Android native code toolchain that seems to change every year.

I'm also returning back to working on MUTA. Currently I'm working on the client hotkey system and UI Lua API. But there's no hurry here, I work on the game when I feel like it.

Oh, and here are some musical picks that I've been playing too much this summer (mostly house and techno, old and new).
Kangding Ray - Summerend
Paul Johnson - Music's In Me
DJ Metatron - U'll Be The King of The Stars
Gianluca Nasci - Stimulation (Stanny Abram Remix)
System F - Dance Valley Theme 2001

Taught Game Development in India for 2 months

1.8.2018 17:11:59

A couple of months ago a friend of mine informed me that my school, Kajaani University of Applied Sciences (KAMK), was going send a delegation to the state of Andhra Pradesh, India, to teach game development for two months. The event would be known as IGDC, short for Indian Game Development Challenge. KAMK was recruiting for the job so I applied and was accepted. I left Finland at the beginning of June and came back home just a day ago, 30th July. I will now attempt to describe the experience in words.

Note that this text is written from my personal perspective and does not represent the views of KAMK nor anyone else who isn't me.

Preparing and leaving for the trip

At some point in time KAMK, my school, had come in contact with APSSDC, the Andhra Pradesh State Skill Development Corporation. They had agreed on KAMK providing the programme for a two month summer course on video game development. Initially the idea had been about bringing Indian students to Finland for the summer, but this was soon diverted so that instead of sending Indians to Finland, KAMK would be sending Finns to India as teachers and organizers. A friend of mine asked me to apply to be part of the delegation going to India and I was accepted. The job was to teach and coach students at the SRM University, Amaravathi.

I don't remember when exactly I got to know I was going to India, but at most I believe it was two months or so before the actual date of leaving. I was working 8 hours a day on an engineering project at that point, so I had little time left for preparing course material or the like. The most work I got to do regarding the India project was when me and my colleague were asked to give an online presentation to some of the Indian students along with a questions and answers session. We staid at work until 9 PM making a PowerPoint because we were told the presentation would need to happen the next day at 7:30 AM. I have to say at this point that little to no time for preparation was sort of a defining feature of this trip. Anyway, the Q&A session left me with one idea on the top of the pile: I would have to work on my understanding of Indian English accents. However, it was also nice to get to know how excited some of the students already seemed.

During the same week we did the Q&A session, three people from India came to visit us in Kajaani, Finland. Two of them were from the APSSDC and one represented SRM University, the school where the course would be held. The Indians were taken hiking as well as to the sauna. Later I heard they were also taken to to see a Finnish dump pit (amongst various other things) which I thought was funny as hell. This was my first contact with Indian people, but also the first time I got to talk to our Finnish project coordinator, an all-around nice guy who would play a very important role during our stay in AP.

Finally at the beginning of June 2018 it was time to leave. I left with one other person only instead of the whole team due to various reasons I will not go into now. Our coordinator, mentioned in the above paragraph, was already waiting for us at SRM University Amaravathi, the campus we would spend the next two months at.

Arrival

We arrived near the weekend and soon met with our own man on the spot, as well as the faculty of the school. We got to see the equipment, which seemed satisfactory, but more computers would be needed - we were going to have nearly 500 students (the APSSDC took care of this well later). The lack of equipment at this point (I assume) had to do with the fact the whole school was still being built. In fact, being built is what was happening to the whole city of Amaravathi 24/7 around the university. The story goes, a couple of years ago the state had broken up into two states and the old capital, Hyderabad, staid in the other state (Telangana), where as Andhra Pradesh was going to build it's own capital from scratch. Indeed, every hour of the day the construction of the school and the city were on-going (which did not help our night sleep at all, by the way).

For our first Sunday (the working week is 6 days in India) one of our local contacts, did something very nice and took us to see his home as well as some other places nearby. That was a gesture we very much appreciated as we got a glimpse of what the country looked like outside of the borders of the campus. The area we were staying at was quite rural, so traveling on the muddy roads took a lot of time. We went to a beach as well and were probably the only people there with nothing but boxers on - I learned later Indians swim in full clothing. Of course we also went to a small temple and ate well.

The first week went on fast. I taught the students some basic concepts of game programming, like what is a program main loop, what tools can you use, etc. as well as practical game programming in C with SDL2. We also talked about version control using Git. A lot of information had to be packed into a very short period of time, and I had not had the time to really prepare any materials at home. I still regret the lack of prepared material, but many people seemed to already catch on and start working on their games. I spent a lot of time in the evenings later preparing course materials for the future since I had it fresh on my mind what was going to be needed.

A week later the rest of our team of 8 Finns arrived. Something that made the first weeks rough for us all was that students did not arrive in one bunch but at the rate of about 40 per day. The total amount at the end was about 480. This meant many things had to be taught again and again, often separately to only a few students at a time. Luckily Indians are (from my limited experience at least) social people and often the students would by themselves inform any new arrivals about various important things.

Course structure

Why would a group of Finnish people be sent to India to teach game development? My understanding is that our job was to introduce the students (and the faculty) to a more practical way of learning, since Indian education is still quite conservative, often (again, to my understanding at least) being about listening to the teacher babble while taking notes. Well, our course was certainly all about practice and not about listening to the teacher babble.

We did some lecturing at the beginning in the form if intensive courses, but just like KAMK game dev summer courses in Finland, most of the time the students worked on their game projects in teams of 4-10 using the tools they wanted. Us teachers, we were simply there available for any questions (or queries, as Indians say) they came up with, occasionally roaming around and checking out the progress being made. People caught onto this model surprisingly quickly, although I have some doubts that there were people even at the end of the course who would have preferred a more theoretical approach - different styles suit different people.

Cultural differences

For someone like me who's never been outside of Europe, seeing even a limited part of India is an eye-opening experience. Just like the EU, India consists of various states with different cultures and languages, but being an ignoramus I had little prior knowledge about how the nation was actually organized. Now I know a little more, but just a little - we only staid in one state (AP), mostly at a university campus, apart from a night or two in Delhi (and that in the near vicinity of the airport - the furthest we got out of the hotel was the liquor store).

Still regarding culture, it warms my heart to see a nation not yet completely taken over by American pop culture (Finland is just about a lost cause at this). Of course many (especially men) wear western clothing and some people know the big Hollywood or pop EDM hits, etc. But at least in the rural areas it seems that, despite the massive presence of American companies such as Coca Cola, Pepsi and the like, people still have their own style of clothing, music, films and customs. That is refreshing and gives some hope for the future of this planet to me.

To just shortly list some differences between Finnish and Indian culture: friendliness (Finnish people don't smile at strangers that much); respect (a nobody like me will get called sir or madam in India, especially wile wearing a blazer); respect for other's time (Indians have little to no sense of punctuality); coffee (Indians have to forcefully put milk and sugar into it). I also cannot help but quote a colleague of mine: how do Indians ever get anywhere by walking so slow. And yet, we all share the same basic needs and understanding of human ethics.

Improvements for the future

This summer course was the first in what is hoped to be many in the near future. Because of that our team had to learn many things in practice. Here are some of those things.

Don't make the course a competition

Since the beginning it was planned that we would choose one team from the course who would be sent to the Slush business event in Helsinki, Finland to pitch their project and company. Immediately upon hearing about this I felt it was a bad idea as it would pitch students against one another. And I feel like that exactly happened, especially towards the end. Games are a business but at least in Finland companies tend to collaborate and share information - ruthless competition isn't that big a part of the picture, or at least that's the feeling I get. Thinking of your co-learner or colleague as a competitor only creates bad blood and burns bridges.

Don't overpromise

We tried hard not to overpromise, but sometimes you should just keep your mouth completely shut instead of telling someone of the possibility of something happening. For example, we were expecting three companies from Kajaani to visit the university, but we should not have spoken about it before the companies confirmed that they were indeed coming. In the end, only one of the companies paid a visit (my greatest thanks go to Virtual Frontiers for this by the way).

Let all students show off their progress

We had nearly 500 people working on various game projects for two full months. That's a lot of working hours, especially considering the working week is 6 days in India and the working day seems to be of a varying length upwards from 8 hours. A massive effort from some very dedicated people to say the least.

All project courses at KAMK end in a post mortem session where students get to dissect everything that went right or wrong during the project period. This way students get to learn from others' mistakes and successes, but at the same time, a post mortem acts as a public revelation of a student project to the public - finally you get to show off the hard work you've done, and usually you also get some useful commentary! The post mortem event is one of the highlights of a course.

Because of scheduling problems, our own mistakes in management, travel timings, stress and various other excuses, we forgot about the post mortems at this event. That was a mistake that I feel made many students feel disappointed, to feel as if their work was not appreciated. That still bothers me. When you've made something, you want to tell the world, and you should be given the chance to do that. And of course, we would have wanted to see those games presented, too.

Keep on pitching

When a project course begins at KAMK, students must first pitch their projects to teachers and get them accepted. We did the same thing at the IGDC, but also kept on holding practice pitching sessions every Friday for the full duration of the course. That was great and should be continued - it was really motivating to see the improvements in the presentations made by the students. Some definite performers there.

Regards

All in all, for me the overwhelming feeling left of this trip is positivity. My heartfelt thanks go to the motivated and friendly students, to the helpful faculty of the organizations involved as well as to the various other great people I (we) met on the way.

The greatest motivator for me was always the progress and enthusiasm of the students who I would thank all personally did I have the time. Here's to hoping for more collaboration between Indian and Finnish educational agencies, and for more events of this kind in the future.

MUTA devlog 3: Lua scripting, game data files and packetwriter improvements

12.5.2018 11:18:45

My most recent workings on MUTA have been about adding ways to implement game content. I have now added the first version of the server side scripting language, Lua. In relation to this work, I've had to implement multiple data file formats so that we know thing slike which scripts to load and which game object uses which script.

File formats for defining game objects

The first thing I wanted to try scripting with were creatures. But before I could script creatures, we had to be able to define them. For that a file format was required. I had used ini-type files so far for quick debug purposes, but I decided it was time to implement a better format.

I've written earlier about the binary mdb file format I wrote for use in the game - we use it for art assets. How ever, while debugging and trying out new stuff, binary files with special editors (mdb has a command line editor in which you give commands like "set column") are a little cumbersome. Text files on the other hand are easy to modify on the go. So a text-based format seemed logical for something like creature definitions that - entries that would change frequently.

Definitions like these aren't complex to represent. An ini file would not do it, but something very close to it would. So here's how our creature definition files look like now:

creature: matlock_creature_1
   name            = Matlock
   description     = Tiesin heti et tollanen haluun ison olla
   sex             = 0.5
   attackable      = 0
   texture_name    =
   script          = test

creature: human_male_1
   name            = Man
   description     = A man
   sex             = 0
   attackable      = 0
   script          = test
   #ae_set is an entity animation set (this line is a comment)
   ae_set          = human_male_1

A similar format is handy for a lot of things: creature, dynamic object, static object and player race definitions to name some. So there's a generalized parser function:

int
parse_def_file(const char *fp,
   int (*on_def)(void *ctx, const char *def, const char *val),
   int (*on_opt)(void *ctx, const char *opt, const char *val),
   void *ctx);

As the function parses the file given in parameter fp, the on_def callback is called when a new definition beginning is found, for example "creature: human_male_1". The callback on_opt is called when an ini-like option is encountered. Ctx is an optional pointer to a caller-defined context. The option name and the value are passed as strings to the callbacks. A missing feature for now are multi-line values, which we might need at some point for longer pieces of text like object descriptions. It's not fancy (actually it's almost the same as .desktop files on Linux or many other similar formats), but these files are both, easy to change on the go and human readable.

Lua scripting creatures

The addition of a scripting language is something I've been meaning to get to for a longer while, but everytime I've meant to get to it, I've found some other feature missing and wandered off. A couple of weeks back I finally got my hands dirty with Lua.

I had not used Lua earlier so it took a little bit of reading the reference manual and the Programming in Lua text found online to figure out how to use the thing. Lua is of course one of, if not
the
most popular embedded scripting language, so the process was fairly simple - if it wasn't, I doubt the language would have such a following.

The first thing I decided to apply scripting for were creatures. Here's how a creature script might look like at the moment:

local v = 0

function Init(creature)
   v = 0
end

function Tick(creature, delta)
   --creature_walk walks a creature in the given direction (0-7)
   creature_walk(creature, v);
   v = (v + 1) % 8
end

The Init() function is called when the script it attached to an entity. The Tick function is called every frame if it exists. As more use cases come up, more functions will probably be added. All of the functions will probably be optional, too, so if an entity doesn't require a script function call every frame, the Tick function for example could be omitted.

The MUTA Lua-side API will probably consist of mostly C functions bound to Lua. In the example above for example, creature_walk is a C function binding. Functions like these operate directly on raw entity pointers (Lua's 'user data' type). I don't know yet if that's a good idea, but it's certainly easy. One thing that must be kept in mind is that since our world will be divided into multiple 'worldd' processes, each simulating a part of the map, the scripts must be movable from one node to another and back again.

Defining scripts

All scripts to be loaded at server start-up must be defined in an ini file. Each line of the file defines the name and the file path of a script, for instance, "test = muta-data/server/scripts/test.lua". This associates an easy-to-remember string id with the script. The given name can then be used in game object definition files to refer to the script.

Script hot reloading

The nice thing about scripts is that you can modify them as the program is running. The way this functionality works in MUTA is that a character with the game master status calls a chat command with the script's name.

When a GM types in, '.reload_script test', a command is sent to every worldd process connected to the master server to reload the script 'test'.  Once the worldd's receive the command, they will look up the script, reload it's Lua code and re-register it's functions in Lua's own registry, then find every entity in the world using the script and reload it for them. That's quite nice for fast iteration!

Packetwriter improvements

Elsewhere, Lommi has been working on improving MUTA Packetwriter, the program we use to define and generate code for network packets. The program works so that every packet in the protocol is defined in a .mp file, which is passed as a parameter to the command-line packetwriter program. The program reads the file and forms the correct structs and inline functions for handling the defined packets, writing them into a C header file.  While the changes aren't upstream yet, there's a bunch of nice stuff incoming.

The improvements include packet definition formatting that isn't as strict as it used to be. It used to be that a white space or linebreak in the wrong place could break the generation but the program would not tell you want went wrong, instead just generating broken code.

You're also now allowed to do arithmetic when defining packet specific size limits.  This is a handy feature for packets that contain variable size elements such as strings.

Support for arrays of packed structs is also on it's way. This is a feature I've been wanting a lot because it's damn handy. For example when sending an account's character list, instead of being able to send an array of structs I've had to create an individual array for every parameter of every character. This has resulted in a lot of friction because populating multiple arrays is obviously more complex than populating a single array, and since our way of using packets is fairly low level, there have been bugs with uninitialized variables and such that took a while to debug.

Up next

I'm currently working on improving creatures. I'm not quite sure what exactly I'll be doing next, but it will be about making gameplay implementation easier, be it more Lua API functionality or actual game object interaction between the client and the server. Better get coding right now!

Ultima Online 20 year post mortem online from GDC 2018

3.4.2018 21:05:14

This is really just a heads up for anyone who missed it: this year's GDC featured a new post mortem session for the now 20 year old Ultima Online, and the video is already available for free here.

UO remains a source of interesting MMO history and design topics to this day. Although the session featured many of the same talking points as the post mortem in 2012, there was a little bit of new stuff there, too. For example, this time Richard Garriott himself took part, and Rich Vogel talked about the invention of game time cards.

I only wish I could find an interview with Rick Delashmit, the original main programmer of the game. It would be interesting to know for example how they stored world state information and account data on disk back in the day!

MUTA devlog 2: cleaning up, async asset loading, GUI memory, creatures

30.3.2018 16:47:24

Alright, here's another post about MUTA, our MMORPG project. I didn't work much on the game for nearly two months since the new year holidays, but I've since been picking up the pace, working the hours I could afford. Unfortunately, it's not that much, but evenings are still something as are weekends. The days are also getting longer and longer, and I've noticed in the past my productivity drops drastically in the middle of the winter when there's only a couple of hours of daylight. Thank goodness spring seems to be making it's way swiftly!

Since Christmas, I believe I alone have touched the codebase. This has a clear advantage: there's no way to get merge conflicts if you only merge from one branch. So, I have been cleaning up here and there, including in files I didn't previously touch personally (although I didn't do anything drastic in the case of such files.), reformatting, clarifying naming conventions, etc. I also implemented asynchronous asset loading proper and started adding support for creatures, but more on that later.

Getting rid of unneeded code

Something I have been wanting to do is getting rid of code that isn't needed. Obviously, the smaller, the better when it comes to codebases - I think we can all agree on that. And we did have some code that was practically duplicate.

For example, we originally had a hashtable implementation that preserved memory addresses and another hashtable that didn't. In the case of the first container, instead of increasing the amount of buckets when more space was needed, a new bucket was created and added to a linked list of buckets at index N. Now, that's not great algorithmically because obviously search time increases linearly with the amount of items in a bucket. But I thought it would be handy when you wanted to store items instead of pointers inside a hashtable, and also wanted pointers to those items to remain valid until the bitter end, even if the table was resized.

Well, turns out the first hash table was only used in one file, in our asset system, and even there the only reason for it's use was that this table was implemented first, back when the other hashtable had not been written yet.

So I got rid of the memory address preserving hashtable.

Less macros

Our containers, including the hashtables, have previously been implemented as C++-template-like C preprocessor macros - macros that define a new struct or structs and functions that manipulate said struct. Hence, to be able to use a container for a specific type, you have to first declare it by calling a macro, like this:

DYNAMIC_HASH_TABLE_DEFINITION(str_int_table, int, const char *, uint32,    fnv_hash32_from_str, 4);
That's kind of cumbersome and not pretty. It's also difficult to read where the struct/function definition happens for anyone new to the codebase. So I would like to get rid of most of this stuff.

Dynamic arrays

The most common container type, dynamic array (vector in C++ lingo), was also implemented as a macro of this sort. So I decided to go through all of our dynamic arrays defined in this manner and replace them with a stretchy buffer like container, as I recently read Our Machinery was doing.

In case you don't know, a stretcy buffer is a group of C macros you put together to be able to create dynamic arrays on the fly. Instead of declaring specific dynamic array types, you declare a null pointer of the data type your array will contain, and then call macros like 'array_push(some_pointer)' or 'array_count(some_pointer)'. The macros will store the size and capacity of the vector at the beginning of the allocation and assign the pointer to point to the first element. For example:

int *array = 0;
array_push(array, 5);
There are a couple of methods to distinguish between a normal pointer and a stretchy buffer. You could do that by naming the varible or leaving a comment. I have taken to typedeffing the types like this: 'typedef int int_darr_t'.

Stretchy buffers only require a single function, which in our case looks like 'void *darr_ensure_growth_by(void *darr, uint32 grow_by, uint32 item_size)', and a bunch of one-liner macros. That simplifies things nicely, because our previous dynamic array definition macro was over a hundred lines long (it has now been erased), and different array types no longer need to be defined via a specific macro call.

By the way, I also made most dynamic containers have a growth rate of 2, meaning they double in size when they fill up. I thought this to be excessive before, but it turns out many standard libraries do this too. Dynamic strings I gave a growth rate of 1.5. In some places we used to have growth rates as small as 5% which was really bad for small groups of objects, leading to a lot of realloc calls (for example, in our GUI).

Other containers

We have some other container types, too. For example, we have a macro for declaring a dynamic object pool for a type, which allocates multiple objects when it needs to grow and provides a linked list-like interface. And of course, we have hash tables. I didn't do anything to these macros yet, because I wasn't sure if I wanted to, but also because of time. One could build a "typeless" hash table for example, but it would be difficult to choose the correct hash function and bucket size  without declaring them via a macro.

Thinking about dynamic memory allocation

When it comes to dynamic memory allocation, I think I'm somewhere between the extremes of liberal and conservative.  Th C++ way of thinking where everything is allocated dynamically is definitely not for me, and since I only really care about the PC as a platform, I also don't feel like I need to know the amount of memory I need beforehand (although I've done that out of curiosity before). The way I would describe my dynamic memory usage is: if it can be allocated statically, allocate it statically. Do that also when you know the absolute maximum amount of bytes an allocation may require. But it's ok to use dynamic arrays and the like when you really don't know.

That being said, I've been pretty liberal about freeing memory. I always let dynamic arrays and similar containers grow until the end of the program without freeing them in between, because I know at some point they will reach their absolute maximum. But I also have not gotten used to cleaning allocations up at the end of a program, since modern operating systems take care of that for you.

I've taken a step back in this thinking though and decided dynamically allocated recources must still be freed at the end of the program, never mind if the operating system is magically capable of doing that for you or not. That' because in larger programs, tracking memory leaks becomes a bit of a pain if you never clean anything up. Tools that track allocations, like AddressSanitizer and Valgrind, will report all leaks, and if you never free anything, the output won't really tell you much because there's a lot of noise.

So far I have added memory clean up to the MUTA client. Now if there is a leak, at least I will be able to tell where it is. I didn't give the same treatment to the server side programs yet however because the whole server side infrastructure will change drastically as the project goes on.

The case of GUI memory management

A sort of memory-related issue I also dealt with cleaning up was the way our GUI's memory was handled. This was due to an experiment I wanted to make when I first wrote the UI base.

MUTA's UI is an immediate mode style one. In the unlikely case you haven't heard of such a thing, take a look at Casey Muratori's  introduction or the popular Dear ImGui library. In short, instead of creating gui "objects", you call functions like this:

gui_begin_window("my window", 16, 16, 64, 64);
if (gui_button(BUTTON_ID, 32, 32, 32, 32))
    do_something();
gui_end_window();
So, it becomes a little like writing HTML when you write, say, inner windows or such. The flow is easy to follow. Anyway, immediate or retained mode, a UI needs to store state, and state storage is what I've been cleaning up.

Have you ever though of what would happen if all your GUI memory was stored in a single, contiguous dynamic block? No? I have. I don't know what was going through my mind when I decided to implement MUTA's UI memory management in this fashion, but I was probably thinking of serialization and to a lesser extent, vague ideas of "cache coherence".

You know what happens when a dynamic memory block is resized? Pointers into it are invalidated if the block has to be moved. Storing a complex structure with many pointers like this may not be the best idea. But that's what I did.

What this lead to was that all pointers had to be stored as relative pointers, integer offsets into the main block of UI memory. This block contained everything, including many dynamic arrays. When a dynamic array had to be resized, we looked in the block if there was enough free space - if not, the whole block had to be resized.

It was a real pain, because you couldn't reliably use pointers even inside functions. If you had a pointer to, say, a window structure, and after fetching that pointer called a function that might move the main memory block of the UI (for example, by drawing some vertices), you had to fetch the pointer again using the aforementioned relative pointer. It was painful to me, and it was more painful to other people who tried touching the UI code earlier.

So I did away with all that. Now the UI has normal, individually allocated dynamic arrays. But at the same time, I removed many of the dynamic elements, beacuse they weren't needed. Some elements are now statically allocated, and there's simply a maximum limit to their number. Others, such as vertices and text formatting buffers are still stored dynamically, because it's hard to know the maximum amount of things such as that beforehand, and because I want the UI to be reusable in other projects.

Immediate mode GUIs are nothing new but I kind of want to write a little about MUTA's specific implementation at some point. Maybe I will, though there are still some features that need adding.

I've uploaded the current GUI API and implementation files here and here in case you're interested.

Adding creatures and NPCs

Cleaning up is the bulk of what I have done lately, but not everything. I added some messages concerning creatures to our protocol, which is rather easy using the MUTA Packet Writer application written by Lommi. The packet writer let's us define packets in a text file, from which it produces serialiazation functions and struct definitions.

So far I only added spawning and despawning, and made sure those work with interest areas (structures that define to whom information of nearby events is sent). Plus a GM command for spawning creatures to test things. But the system is surprisingly complicated to implement, because we will need a concept of what spawn points are, how creature IDs are handled, etc. We don't just want to spawn creatures into the world without any context - that may lead to bugs like unique NPCs spawning twice, or something worse. We also need a format to store those spawn points in, and lastly we're going to need scripting. The latter will probably be done with Lua.

Asynchronous asset loading

The asset system has seen some grand internal upgrades. Textures and sounds may now both be loaded asynchornously. This was possible before, but only in a half-assed, buggy kind of way. There's still work to be done here, but it kind of works.  There's also a sort of garbage collection mechanism in place which gets rid of unused assets after a while.

The asset API remains about the same as before. It looks something like this:

tex_asset_t *
as_claim_tex_by_name(const char *name, bool32 async);

tex_asset_t *
as_claim_tex_by_id(uint32 id, bool32 async);

void
as_unclaim_tex(tex_asset_t *tex);

Clean up takes time, perfection is impossible

Everyone in software knows this, but there is no such thing as perfect code. There's always something to improve upon. And that's definitely something I came to think of once again while going through our codebase.

There are so many things I'd wish to do better. I want to clean up formatting, I want to touch this and that irrelevant "issue" because I know there's a more optimal solution. For example, I could do away with some slight branching in the updating of moving objects on the client side by separating the objects into two different arrays: idlers and movers. But this kind of thinking is treacherous. It's a 2D game, with only up to maybe a thousand mobile objects on screen at once. It just doesn't matter. But the engineer inside most programmers craves to take on the most minimal of issues.

Beautifying code is as treacherous as over-engineering. I could spend years making the code prettier, but if it does not significantly reduce the amount of time it takes to upkeep said code, it just isn't worth it. So, in the near future, MUTA's codebase will retain it's ugly macros and similar beauty flaws.

When the urge strikes to spend time on something that isn't worth it, I try to think of existing games. I'm certain no successful game has ever shipped without it's programmers thinking they could've done a lot better if only they'd had a little more time. Especially I think of Ultima Online, which was one of the first graphical MMOs and groundbreaking in a lot of ways, and yet most of it, according to Raph Koster, was written by a single programmer over the span of as little as three years. Now that's one heroic feat even from  a larger team of programmers, but it also signals to me that the codebase must have been pretty damn far from perfect.

On Finland deporting Russian diplomats

28.3.2018 08:41:25

Note: this post contains personal opinions reflective only of the author's views and not necessarily of anyone else who partakes in the upkeep of this website.

Often times nowadays, politicians in democratic societies appear as if they have completely abandoned ideology. Ideology is often frowned upon, at least if it isn't of the kind that emphasizes the growth of the markets, increasing the amount of foreign investments in the country, or something in that fashion. But emphasizing these points, of course, is not viewed as ideology at all - it is merely viewed as pragmatism through the narrow lens of market centricism.

But sometimes ideology still makes an appearance. You've probably heard of many EU countries deporting Russian diplomats because of the poisoning of Sergei Skripal in Britain, an incident the culprit of, the UK and US claim, is Russia. Finland is amongst those countries.

Now, I don't know who poisoned Skripal. But I believe that neither do Finnish officials. All they have, at least so I believe, is the word of two NATO countries, the US and the UK. And how trustworthy exactly is that word?

In the traditional nationalist Russia-fearing Finnish mindset where everything that comes from the west is superior, held on to by a certain portion of the population, the word of said countries is apparently very trustworthy, at least if it's an excuse to provoke the grand eastern neighbour (which is rather analogous to playing with fire). Never you mind the UK only gave Russia a single day to respond to the accusations they made, or the fact no evidence of Russian interference has been released to the public.

I feel as if Finnish politicians, those who are afraid of Russia at least, are forgetting what the best strategy has always been for "defense" in this small country between two great powers: neutrality. And diplomacy. Not only are we now making decisions based on no evidence, but we're risking our eastern relationship in the name of the ideology of West is Best, and not even for the first time.

Of course, if it is proven Russia was behind the attack, it's a different matter. But this hasn't happened, not so far at least.

Not rewriting it in Rust, and on Unity's C# source release

27.3.2018 10:04:43

Ain't not rewriting it in Rust

Who hasn't heard the phrase "rewrite it in Rust"? I certainly have, and yesterday finally thought, "you know, maybe I should give the language a try - a safer version of C sounds pretty tempting". So I installed rustc, opened a quick tutorial from the web and dived in at the shallow end.

I got as far as structs in about 5 minutes. So off I went and declared my struct:

struct s {
   x: i32,
   y: i32
}
All well and good I thought, but the compiler decided to be nasty. It told me:
warning: type `s` should have a camel case name such as `S`

I'm sorry, but I don't think its the job of the language to tell me how to name my data structures or variables, other than for things like whether they can include spaces/numbers or not, etc. I can go along with indentation marking scope, like in Python or other languages - I believe that's been proven to improve readability across the board. But I'm not interested in styling my naming conventions after the personal tastes of the creator of the language.

I realize styling warnings can be disabled via compiler options, but if its in the standard, you probably shouldn't. So I guess that's it for Rust from my part for now.

Unity's C# source code reference-only release

Unity recently decided to release it's engine and editor C# source code. The C++ core of the engine remains out of sight, and even the C# release is made under a reference-only license, meaning that modifying the source is not allowed. So, unfortunately we don't come even close to fulfilling the four software freedoms as traditionally defined. Unity seems to stress this themselves in the blog post. And of course, one may argue this means little since the C# source code could already be decompiled by parties interested. But I won't deny it's still a very small step in the right direction.

The release, I think, is not that big of a deal though. Unortunately, Unity still appears to hold some old-fashioned views popular in the proprietary software sphere, as proven by this statement, taken directly from the Unity blog: Wed open source all of Unity today if we thought we could get away with it and still be in business tomorrow.

I don't think that statement makes much sense. Of course taking a more open approach would require slight adjustments in the way the company does business, but Unity is already in the market of software as a service. It's not just the engine itself, its the documentation and updates people want. Even if they had a fairly open license that allowed reading, modifying and distributing source code, Unity could keep on raking in the money from the 5% they charge from people who make games built with their engine. The source being open would, I believe, really have no negative effects from this point of view.  Also, knowing Unity's limitations (which are many), I'm fairly sure there's no such advanced technology inside the engine that it would do much for a competitor to read the code, especially considering their main competitor, Unreal Engine 4, already has it's source visible.

Well, here's to hoping software vendors slowly come to their senses. Free/open source is not only ethical, but also practical - knowing your underlying technology from the inside out certainly makes it easier to make a good game.

Page: 0 1