SilkTouch: Invokes & Marshalling

SilkTouch? What even is that?

Most people don't realize this, but a lot is going into getting all those bindings going. SilkTouch is our solution to automated marshalling. Let's first look into what our bindings pipeline looks like, and then I'll go into more detail what SilkTouch is, what it does, how it works.

BuildTools

This isn't really my area, but I want to briefly touch on it. BuildTools is what takes a huge configuration file linking specifications and headers, and turns it into signatures. That is it's primary purpose, producing signatures. It saves those as partial C# classes, cause this takes, in comparison, a lot of time. BuildTools is a great tool for automating API updates, but that's a topic for another blog post.

The most basic case

SilkTouch is a Source Generator (SG), source generators are a C# 9 feature which and are essentially analyzers, just they get to contribute some code, they get the full compilation (so all code + SyntaxNodes + Roslyn tools) and then get to work. First of all, SilkTouch is a pretty complex SG, and some would say it abuses what SGs are supposed to do. First of all, it takes exceptionally long to complete (not BuildTools long, but a lot longer than your average SG), and instead of generating (C#) strings like most generators, it uses Roslyn SyntaxNodes / SyntaxFactory to generate code, so don't develop a fear for SGs if you look at SilkTouch. They are amazing, I promise!

Now, above I've said "SilkTouch is our solution to automated marshalling". Our Core (Silk.NET.Core) is what does all the native library loading, but it only gives us the raw native symbol, for example (delegate unmanaged[Cdecl]<uint, uint, GLEnum, GLEnum*, uint*, GLEnum*, uint*, byte*, uint>) (this is the function pointer form introduced in C# 9, Silk.NET.Core will actually give you an IntPtr). Now, SilkTouch doesn't really know this either, it will (usually) only get something like

[NativeApi(EntryPoint = "glGetDebugMessageLog")]
public unsafe partial uint GetDebugMessageLog(
    uint count,
    uint bufSize,
    [Count(Parameter = "count")] GLEnum* sources,
    [Count(Parameter = "count")] GLEnum* types,
    [Count(Parameter = "count")] uint* ids, 
    [Count(Parameter = "count")] GLEnum* severities, 
    [Count(Parameter = "count")] uint* lengths, 
    [Count(Parameter = "bufSize")] byte* messageLog);

This is the most basic case, for now. Here, all the parameters are already in their native form. SilkTouch won't do all that much in this case, the parameters will just pass through the marshalling pipeline, which should feel somewhat familiar to anyone who used Kestrels (ASP.NET Cores) Middlewares. It will then end up at the terminal middleware, called BuildLoadInvoke as you may figure, this build the native invocation call. This will essentially build something like

n8 = ((delegate *unmanaged[Cdecl]<uint, uint, GLEnum*, GLEnum*, uint*, GLEnum*, uint*, byte*, uint>)(CurrentVTable as GeneratedVTable).Load(719, "glGetDebugMessageLog"))((count), (bufSize), (sources), (types), (ids), (severities), (lengths), (messageLog));

A function pointer signature generated by SilkTouch. Simplified from the original for readability.

You may notice that this includes a cast to the previously mentioned native signature, and then a call of said signature using the method parameters. We'll for now ignore the (CurrentVTable as GeneratedVTable).Load(719, "glGetDebugMessageLog") bit, I'll talk about that down the road.

Awesome! We've made it, that really is all there is to it, in this most basic case. Here is the full signature again:

public unsafe partial uint GetDebugMessageLog(uint count, uint bufSize, GLEnum* sources, GLEnum* types, uint* ids, GLEnum* severities, uint* lengths, byte* messageLog)
{
	uint n8;
	n8 = ((delegate *unmanaged[Cdecl]<uint, uint, GLEnum*, GLEnum*, uint*, GLEnum*, uint*, byte*, uint>)(CurrentVTable as GeneratedVTable).Load(719, "glGetDebugMessageLog"))((count), (bufSize), (sources), (types), (ids), (severities), (lengths), (messageLog));
	return n8;
}

A finished function generated by SilkTouch. Simplified from the original for readability.

Marshalling

Now, we don't want to make our users do all the unsafe stuff themselves, for example, users don't have to handle strings themselves, we do our best to handle that. One such overload is

[NativeApi(EntryPoint = "glGetDebugMessageLog")]
public unsafe partial uint GetDebugMessageLog(
    uint count,
    uint bufSize,
    [Count(Parameter = "count")] GLEnum* sources,
    [Count(Parameter = "count")] GLEnum* types,
    [Count(Parameter = "count")] uint* ids,
    [Count(Parameter = "count")] GLEnum* severities,
    [Count(Parameter = "count")] uint* lengths,
    [Count(Parameter = "bufSize")] out string messageLog);

This again is what SilkTouch gets from BuildTools. Most of the parameters will just pass through the Marshalling pipeline, but not string! string requires some action from us. The StringMarshaller (the second element in the pipeline) will catch this. Because there are many different ways to encode a string before passing it to native code, it will check whether it knows what this parameter is supposed to be marshalled as (possibilities are BStr, LPWStr, LPStr and LPTStr) or fallback to LPStr.

Once it knows what native type to marhsal the parameter as, it'll check for the RefKind of the parameter. In this case, out. out parameters are significantly different to the other RefKinds because it needs to figure out how much memory to allocate. We do this using the Count attribute. In this instance a parameter is specified, so we'll pass that parameter as the length argument to SilkMarshal.AllocateString to allocate the buffer for the native string. After all of that, at long last we have a pointer we can pass to native!

Awesome, the rest of the parameters are fine as is, and we normally generate our invoke like above:

n9 = ((delegate *unmanaged[Cdecl]<uint, uint, GLEnum*, GLEnum*, uint*, GLEnum*, uint*, byte*, uint>)(CurrentVTable as GeneratedVTable).Load(721, "glGetDebugMessageLog"))((count), (bufSize), (sources), (types), (ids), (severities), (lengths), n8);

Simplified from the original for readability.

Now, the way the Pipeline works is heavily inspired by how the ASP.NET Core Middlewares work. The first middleware is called with the context, and a callback for the next middleware, and so on. The terminal middleware (BuildLoadInvoke) just does not call next() (doing so would throw) and the recursive stack resolves, now each middleware gets the chance to do post-invoke processing. The string middleware has remembered that this parameter was out and needs some readback. This is done using SilkMarshal.PtrToString the result is written into messageLog and the string buffer is freed using FreeString we return, and done. Here is the full version again:

public unsafe partial uint GetDebugMessageLog(uint count, uint bufSize, GLEnum* sources, GLEnum* types, uint* ids, GLEnum* severities, uint* lengths, out string messageLog)
{
	uint n9;
	byte* n8;
	
	n8 = (byte*)SilkMarshal.AllocateString((int)bufSize, (NativeStringEncoding)20);
	
	n9 = ((delegate *unmanaged[Cdecl]<uint, uint, GLEnum*, GLEnum*, uint*, GLEnum*, uint*, byte*, uint>)(CurrentVTable as GeneratedVTable).Load(721, "glGetDebugMessageLog"))((count), (bufSize), (sources), (types), (ids), (severities), (lengths), n8);
	
	messageLog = SilkMarshal.PtrToString((IntPtr)n8, (NativeStringEncoding)20);
	SilkMarshal.FreeString((IntPtr)n8, (NativeStringEncoding)20);
	return n9;
}

Simplified from the original for readability.

If you are interested in this kind of thing, I encourage you to have a look at https://github.com/dotnet/Silk.NET/tree/v2.0.0/src/Core/Silk.NET.SilkTouch/Middlewares It's not that bad, I promise, especially the GenericPointerMarshaller is super simple. Onwards in another Blog Post, about Slots & VTables (the bit we ignored above, (CurrentVTable as GeneratedVTable).Load(719, "glGetDebugMessageLog")). If you'd like to find out more about SilkTouch, again take a look at SilkTouch's source code or head over to our Discord server where myself or one of the other maintainers, contributors, or regulars will be happy to answer your questions or just generally have a natter.