Network message serialization in MUTA
MMOs typically have a lot of network message types for different actions: move my character, talk to this NPC, a spell was cast by a unit.... A set of message types along with rules for reading them forms a protocol.
MUTA has many internal communication protocols, all of which run over TCP/IP: client - server, master - simulation, master - login, master - world database, master - proxy... at least. All of these are binary protocols: usually, the first one or two bytes denote the type of the message, after which comes a set of primitive values, such as sized integers, arrays, etc.
This is an example from the master - proxy protocol. This message may be sent
by master to proxy or proxy to master, though most messages in MUTA are one
direction only. It is used in either of the following two scenarios:
- A client sends a message to the master server through the proxy server.
- The master server sends a message to a client through the proxy server.
Packing messages to streams takes code, and writing such code is very errorprone. Thus, MUTA has a tool for this. The original tool, MUTA Packetwriter, was written by Lommi 2 years ago. Recently, I took to rewriting the tool to add a couple of features and modify others, and to make it more maintainable.
The packetwriter is a command line tool that takes in two arguments: an in-file that defines the messages in a protocol, and an out-file into which to write the generated C code. The in-file format uses the same .def format we use all around MUTA.
__group = svmsg
id = uint64
name = char <uint8> {MIN_CHARACTER_NAME_LEN, MAX_CHARACTER_NAME_LEN}
race = uint8
sex = uint8
map_id = uint32
Above is an example of a message definition in the in-file of the client - server protocol. The message belongs to the group svmsg, which is a group defined in the same file. If this message is the third svmsg-group message definition file in the file, it's ID will be 3 - this enumeration is what groups are for.
In the above message, the variable name is a string, the curly braces marking it a variable-size array (of type char). Strings in MUTA's protocols are never null-terminated. Instead, a length variable is written for each array in a message - if the max length is numeric, the packetwriter will automatically figure out the required type of the length variable (uint8, uint16...), but if it's alphabetical, because we can't parse #defines from C files, the type of the length variable can be explicitly marked, which is what we do with the <uint8> notation above. All arrays, strings included, must have a maximum length defined, and can have an optional minimum length.
The packetwriter produces the following code from the above declaration.
{
... other messages ...
SVMSG_NEW_CHARACTER_CREATED,
... other messages ...
};
struct svmsg_new_character_created_t
{
uint64 id;
struct {uint8 len; char data[MAX_CHARACTER_NAME_LEN];} name;
uint8 race;
uint8 sex;
uint32 map_id;
};
static inline int
svmsg_new_character_created_compute_sz(svmsg_new_character_created_t *s);
{
int ret = 0;
ret += 8;
ret += 1 + (int)s->name.len;
ret += 1;
ret += 1;
ret += 4;
return ret;
}
static inline int
svmsg_new_character_created_write(bbuf_t *bb, svmsg_new_character_created_t *s)
{
uint8 *m = bbuf_reserve(bb, 1 + svmsg_new_character_created_compute_sz(s));
muta_assert(m);
pw2_write_uint8(&m, SVMSG_NEW_CHARACTER_CREATED);
muta_assert(s->name.len >= MIN_CHARACTER_NAME_LEN);
muta_assert(s->name.len <= MAX_CHARACTER_NAME_LEN);
pw2_write_uint8(&m, s->name.len);
pw2_write_uint64(&m, s->id);
pw2_write_uint8(&m, s->race);
pw2_write_uint8(&m, s->sex);
pw2_write_uint32(&m, s->map_id);
for (uint8 i = 0; i < s->name.len; ++i)
pw2_write_char(&m, s->name.data[i]);
return 0;
}
static inline int
svmsg_new_character_created_read(bbuf_t *bb, svmsg_new_character_created_t *s)
{
int size = SVMSG_NEW_CHARACTER_CREATED_MIN_SZ;
int free_space = BBUF_FREE_SPACE(bb);
if (size > free_space)
return 1;
uint8 *m = BBUF_CUR_PTR(bb);
pw2_read_uint8(&m, &s->name.len);
if (s->name.len < MIN_CHARACTER_NAME_LEN)
return -1;
if (s->name.len > MAX_CHARACTER_NAME_LEN)
return -2;
size += s->name.len * 1;
size -= (MIN_CHARACTER_NAME_LEN) * 1;
if (size > free_space)
return 2;
pw2_read_uint64(&m, &s->id);
pw2_read_uint8(&m, &s->race);
pw2_read_uint8(&m, &s->sex);
pw2_read_uint32(&m, &s->map_id);
for (uint8 i = 0; i < s->name.len; ++i)
pw2_read_char(&m, &s->name.data[i]);
bb->num_bytes += size;
return 0;
}
To use the generated code above to serialize a message to send it over the network, we just need to fill in a struct and call svmsg_new_character_created_write();
.id = 53,
.race = 1,
.sex = 0,
.map_id = 32};
memcpy(s.name.data, "John", 4);
s.name.len = 4;
/* Byte stream to write to */
uint8 *memory = ...
bbuf_t bb = BBUF_INITIALIZER(memory, svmsg_new_character_created_compute_sz(&s));
svmsg_new_character_created_write(&bb, &s);
The generated return values for the message reading functions have meaning in that a positive return value implies the message was incomplete, a negative return value implies the message was illegal, and a return value of zero implies the message was OK.
The packetwriter also supports structs if they're defined in the in-file. The structs may also be nested, or in arrays, or contain arrays.
Security features are limited to array length checking and numeric variable range checking.
Bitpacking, the act of packing multiple numbers whose legal ranges are known into a single variable as bits, is not supported yet.
That's mostly all worth saying about that subject I guess. You can find the Packetwriter 2 (and for now, also Packetwriter 1) code in the MUTA repository under tools/packetwriter2 in case you're curious. I'm sure there's still bugs out there in the code though.
