Yet another custom binary protocol library implementation

Recently I have been involved in a project that required a custom binary protocol specification be written for interaction between some custom hardware and a central processing server. Before
you wonder why re-invent the wheel I did look into protoBuf but although perfect for what I was doing, implementing this on the custom hardware would have been more difficult.

The communication was going to occur over TCP using sockets. Some of the requirements of this protocol included:

1. A small size protocol to minize network traffic
2. Libraries that could be used across multiple applications to handle the protocol
3. An extendable protocol so that new packet types could be included without altering the core layer
4. A flexible protocol to handle multiple scenarios such as variable string fields
5. The ability to implement backwards compability on packets

Based off these requirements we ended up producing a simple protocol specification that worked at the binary level. Each packet would be deliminated and contain data in the following format:

[stx][identifier][cmd][packet id][parent packet id][timestamp][data payload][crc][etx]

where

where definitions include

Based off this I was able to produce a library that allowed for easy creation of payloads and serializing to and from a byte[]. Below are a few features of this library.

Checksum

A very simple checksum was implemented to ensure basic data integrity. It is noted that this is a particular weak point of the protocol and that it would be more robust with a 2 byte checksum.

However at the time of writing this would require some re-work although defintely worth invesitaging.

public byte GetCheckSum(byte[] data)
{
	int checksum = 0;

	for (int i = 0; i < data.Count(); i++)
	{
		// No. XOR the checksum with this character's value
		checksum ^= data[i];
	}

	return (byte)(checksum & 0x00FF);
}

Byte stuffing

There are two key bytes used in the protocol. $ and *. Because of this we used byte stuffing in order to maintain the integrity of thd data being transmitted.

Packet data is byte stuffed on transmission using the ^ symbol followed by the stuffed byte. They are replaced on receiving by replacing the ^ and + 1 byte with the relevant mapped character. This ensures the integrity of packet keywords.

Byte stuffing is all maintained within two simple classes that enable us to byte stuff and unstuff any data we wish.

var unStuffedData = new byte[] { some bytes };
var stuffer = new BinaryPacketStuffer(ignoreStxEtx: true);
var stuffedData = stuffer.Stuff(unStuffedData);

In some cases the data we are stuffing contains a $ and * in which case we want to ignore those two properties. The boolean parameter ignoreStxEtx allows us to do just that if desired.

Data payloads

Data payloads form the core extensibility of the protocol and is where you can define any type of data specification you desire. An example of a simple payload that would take at minimum 6 bytes and a variable amount of bytes read in as a string.

public class LogRequest : DataPayload
{
	public int ErrorCode { get; set; }
	public byte Length { get; set; }
	public string Text { get; set; }

	public override void Write(IDataOutputStream outputStream)
	{
		outputStream.Write((short)ErrorCode);
		outputStream.Write(Length);
		outputStream.Write(Text, Text.Length);
	}

	public override void Read(IDataInputStream inputStream)
	{
		ErrorCode = inputStream.ReadInt16();
		Length = inputStream.ReadByte();
		Text = inputStream.ReadString();
	}
}

Versioning

Each packet version is identified in the header payload. Because of this we can implemet specific readers for our payloads so that we can handle old versions of a packet as well as new versions without having to worry and contain nasty switch statements throughout our application.

This is implemented in our Payload specific implementation by overriding the CreateVersionReader() method. An example is as follows

protected override IPacketReader CreateVersionReader(int version)
{
	switch (version)
	{
		case 0:
			return base.CreateVersionReader(version);
		default:
			return new AnalogInputChangedReader(this);  // by default use yhe latest known reader                  
	}
}

Creating packets using PacketBuilder

There is a built in class that can easily create packets and allow you to serialize to and from byte[]. This class is called the PacketBuilder and you would use it as such:

Converting to a byte array

var packetBuilder = new PacketBuilder();
var expectedPacket = packetBuilder.CreateEmptyPayload(PacketCommands.AlertRequest);
expectedPacket.Data.TypeOfAlert = AlertRequest.AlertTypes.MainBatteryLost;
expectedPacket.Data.Value = 100;

var bytes = packetBuilder.Build(expectedPacket);

Converting from a byte array

var rawData = new byte[]
					  {
						  0x24,0x01,0x61,0xAB,0x0C,0x04,0x00,0x00,0x00,0x00,
0x00,0x4D,0x34,0x51,0x56,0x04,0x08,0x0F,0xE8,0x32,
0x00,0x08,0x22,0xE5,0x30,0x00,0x08,0x32,0xD2,0x33,
0x00,0x08,0x34,0xC0,0x31,0x00,0x8D,0x2A
					  };

var packetBuilder = new PacketBuilder();
var analogPacket = packetBuilder.Build(rawData);

Code review

Some discussion of my original thoughts around this protocol can be found at:

http://codereview.stackexchange.com/questions/44625/writing-and-reading-of-a-custom-binary-protocol

Known limitations

1. The amount of commands is limited to what can fit into a single byte
2. The checksum is very small and there is currently no way to insert a custom implementation

Some future enhancements

1. I'm considering exending the ability to use attributes and so not need to implement Read and Write for every payload if you decorate your properties with the necessary attributes. Currently this is loosly implemented
but does require some testing (28-March-2016).
2. It would be great to allow custom checksum algorithms and those algorithms to specify the checksum length.
3. Increasing the command field from one byte

Nuget packages

- Symtech.G5.Protocol

Summary

This library has worked well and has produced a solid foundation for sending and receiving data sent as binary. It is easily extendable with new types of Payloads making it flexible as a protocol
library in and of itself. An existing protocol is found as a nuget package under Symtech.G5.Protocol.V2.