(insert ../../shared/header.md.html here)

**<big>Blender as Editor Devlog #1**</big><br/>

<br/>

![Blender Live Link in Action](live_link_video_1.mov)

A couple of months back I watched a great talk from the folks at Santa Monica Studio at REAC 2024. Their talk [Maya as Editor: The game development approach of Santa Monica Studio](https://www.youtube.com/watch?v=ZwPogOhbNWw) outlined how Santa Monica Studio uses Maya for a large portion of their game-engine tooling. I had always liked the idea of leveraging Blender to serve a similar role in my personal projects and figured there's no time like the present to give it a shot.<br/>

Why use an existing 3D application as your editor? First, it's a huge time-saver. If you already have a piece of software that does most of what you require to build out your 3D worlds, then extending that software is a much easier task than creating your own editor from scratch. Second, the method I'll outline below gives you a lot more control over how data is ferried between the two applications. Even if you were to roll your own toolset for designing levels, I suspect most projects will still require some amount of work in another 3D application for modelling, texturing, and animation. Rather than dealing with a bunch of GLTF/FBX files that you import into your level-design toolset, we will treat the .blend files as our single source of truth. We can send an initial full update to the game on init, but then we'll only send smaller incremental updates when anything changes.

**Plan**<br/>

There's two main pieces of software going into this little experiment:
1. A blender extension, which is responsible for sending data from blender to our game 
2. Our game, which listens for updates from our extension and runs any gameplay/simulation logic on that data

I've opted to use the [flatbuffers library](https://github.com/google/flatbuffers) to package up and send data. This allows us to describe our data layout in a spec file and then auto-generate both python and c++ code for reading/writing data. That data will then be sent from blender to our game via sockets. Finally, we'll use the excellent [Sokol libraries](https://github.com/floooh/sokol) to handle rendering the data sent from Blender in our game.

**Data Layout**<br/>

Our first order of business will be to describe the data we need to send. For this first version, we'll only concern ourselves with sending basic object information and static mesh data. For each object we'll send its name, unique_id, visibility, location, scale, and rotation. If the object is a mesh object, we'll also send an optional 'mesh' entry, which contains our vertex and index data for that mesh. Flatbuffers uses an IDL file to describe the data to be written/read in a C-like data-description-language. 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C default
// blender_live_link.fbs
// Flatbuffers IDL file for blender live link
 
namespace Blender.LiveLink;

struct Vec3 {
  x	: float;
  y	: float;
  z	: float;
}

struct Vec4 {
  x	: float;
  y	: float;
  z	: float;
  w	: float;
}

struct Quat {
  x	: float;
  y	: float;
  z	: float;
  w	: float;
}

struct Vertex {
	position	: Vec4;
	normal		: Vec4;
}

table Mesh {
	vertices	: [Vertex];
	indices		: [uint];
}

table Object {
	name		: string;
	unique_id	: int;
	visibility	: bool;

	location	: Vec3;
	scale		: Vec3;
	rotation	: Quat;

	//optional mesh data
	mesh		: Mesh;
}

table Update {
	objects	: [Object];
}

root_type Update;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

After defining our schema file (blender_live_link.fbs) we can run the flatbuffer compiler to generate code for the languages we'll need. We'll need to output python code to write the data in our blender extension and C++ code to read it in our running game: 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ bash default
./flatbuffers/build/flatc -o compiled_schemas/python --python blender_live_link.fbs 
./flatbuffers/build/flatc -o compiled_schemas/cpp --cpp blender_live_link.fbs 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Writing Flatbuffers Data from Blender (Python) **<br/>

Over in our game, we'll always expect the incoming flatbuffer to be our root_type (Update). This update will contain some number of objects we need to either add, change, or remove. Our initial update will send all of the objects in the active blender scene, but we can be more conservative about the updates we send later when only a few objects have changed. Using the python API flatbuffers generated for us, we build up the payload we'll send across the net to our running game.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Python default
# Function that takes in a list of blender objects and deleted object uids 
# and sends their relevant data to our running game
def make_update(self, in_object_list, in_deleted_object_uids):
	# Evaluate Depsgraph
	dependency_graph = bpy.context.evaluated_depsgraph_get()

	# init flatbuffers builder
	builder = flatbuffers.Builder(0)

	# Build up objects to be added to scene objects vector
	live_link_objects = []
	for blender_object in in_object_list: 
		# Only add if enable_live_link is set
		if (blender_object.live_link_settings.enable_live_link):
			live_link_objects.append(
				self.make_flatbuffer_object(builder, blender_object, dependency_graph)
			)

	# actually create the scene objects vector
	Update.UpdateStartObjectsVector(builder, len(live_link_objects))
	for live_link_object in live_link_objects: 
		builder.PrependUOffsetTRelative(live_link_object)
	update_objects = builder.EndVector()

	Update.UpdateStartDeletedObjectUidsVector(builder, len(in_deleted_object_uids))
	for deleted_object_uid in in_deleted_object_uids:
		builder.PrependInt32(deleted_object_uid)
	update_deleted_object_uids = builder.EndVector()

	Update.Start(builder)

	# Add objects vector to scene
	Update.AddObjects(builder, update_objects)
	Update.AddDeletedObjectUids(builder, update_deleted_object_uids)

	# finalize scene flatbuffer
	live_link_scene = Update.End(builder)

	builder.FinishSizePrefixed(live_link_scene)
	
	return builder.Output()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The bulk of the actual data processing happens in `make_flatbuffer_object` If we encounter a mesh object, we make sure to evaluate its modifiers and triangulate it. Then we add its vertices and indices to our flatbuffer.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Python default

def make_flatbuffer_object(self, builder, obj, dependency_graph):
	# Allocate string for object name
	object_name = builder.CreateString(obj.name)

	# Mesh Data
	mesh = None
	if obj.type == 'MESH': 
		# Helper function to evaluate our mesh object's modifiers and triangulate it 
		mesh = self.get_mesh(obj, dependency_graph)

		# Export Vertices
		blender_vertices = mesh.vertices
		
		Mesh.MeshStartVerticesVector(builder, len(blender_vertices))
		for blender_vertex in reversed(blender_vertices):
			position = blender_vertex.co.to_4d()
			normal = blender_vertex.normal.to_4d()
			Vertex.CreateVertex(
				builder,
				position.x, position.y, position.z, position.w,
				normal.x, normal.y, normal.z, normal.w
			)
		mesh_vertices = builder.EndVector()

		# Export Indices
		indices = []
		for blender_polygon in mesh.polygons:
			# Should be able to assume triangles here because of triangulate call above
			indices.append(blender_polygon.vertices[0])
			indices.append(blender_polygon.vertices[1])
			indices.append(blender_polygon.vertices[2])

		Mesh.MeshStartIndicesVector(builder, len(indices))
		for index in reversed(indices):
			builder.PrependUint32(index)
		mesh_indices = builder.EndVector()

		Mesh.Start(builder)
		Mesh.AddVertices(builder, mesh_vertices)
		Mesh.AddIndices(builder, mesh_indices)
		mesh = Mesh.End(builder)
	
	# Begin New Object 
	Object.Start(builder)
	
	# Object Name
	Object.AddName(builder, object_name)

	# Session UID (note that this is a fairly new addition to the python API)
	session_uid = obj.session_uid
	Object.AddUniqueId(builder, session_uid)

	is_visible = obj.visible_get()
	Object.AddVisibility(builder, is_visible)

	# Object Location
	location_vec3 = Vec3.CreateVec3(builder, obj.location.x, obj.location.y, obj.location.z)
	Object.AddLocation(builder, location_vec3)

	# Object Scale
	scale_vec3 = Vec3.CreateVec3(builder, obj.scale.x, obj.scale.y, obj.scale.z)
	Object.AddScale(builder, scale_vec3)

	# Object Rotation
	rot = obj.rotation_euler.to_quaternion()
	rotation_quat = Quat.CreateQuat(builder, rot.x, rot.y, rot.z, rot.w)
	Object.AddRotation(builder, rotation_quat)

	# Add Object Mesh Data if it exists
	if mesh != None:
		Object.AddMesh(builder, mesh)

	# Add Rigid Body Data if it exists
	if obj.rigid_body:
		Object.AddRigidBody(builder, RigidBody.CreateRigidBody(
			builder, 
			isDynamic = obj.rigid_body.enabled,
			mass = obj.rigid_body.mass
		))

	# End New Object add add to array
	live_link_object = Object.End(builder)

	return live_link_object

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Reading Flatbuffers Data in our Running Game (C++) **<br/>

In our currently-running game, we'll use the generated C++ flatbuffers code to read the incoming data that we wrote and sent in our blender extension.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C++ Default
// Live Link Function. Runs on its own thread
void live_link_thread_function()
{
	// Init socket we'll use to talk to blender
	struct addrinfo hints, *res;
	// first, load up address structs with getaddrinfo():

	memset(&hints, 0, sizeof hints);
	hints.ai_family = AF_UNSPEC;
	hints.ai_socktype = SOCK_STREAM;

	// Connection info to connect to blender on localhost
	const char* HOST = "127.0.0.1";
	const char* PORT = "65432";
	getaddrinfo(HOST, PORT, &hints, &res);

	// make a socket
	state.blender_socket = socket(res->ai_family, res->ai_socktype, res->ai_protocol);
	bind(state.blender_socket, res->ai_addr, res->ai_addrlen);
	const i32 backlog = 1;
	listen(state.blender_socket, backlog);

	// accept connection from blender
	struct sockaddr_storage their_addr;
	socklen_t addr_size = sizeof their_addr;
	int connection_socket = accept(state.blender_socket, (struct sockaddr *) &their_addr, &addr_size);

	// infinite recv loop
	while (true)
	{
		sbuffer(u8) flatbuffer_data = nullptr;

		int current_bytes_read = 0;
		int total_bytes_read = 0;
		int packets_read = 0;
		optional<flatbuffers::uoffset_t> flatbuffer_size;
		do 
		{
			const size_t buffer_len = 4096; 
			u8 buffer[buffer_len];
			const int flags = 0;
			current_bytes_read = recv(connection_socket, buffer, buffer_len, flags);

			// Less than zero is an error
			if (current_bytes_read < 0)
			{
				printf("recv_error: %s\n", strerror(errno));
				exit(0);
			}

			// No bytes read this iteration. Try again
			if (current_bytes_read == 0)
		  	{
				continue;
			}

			// current_bytes_read > 0, we've got data!
			if (current_bytes_read > 0)
		  	{
				// Flatbuffer size will be prefixed to flatbuffer data. Set it when we encounter it
				if (!flatbuffer_size)
				{
					assert(current_bytes_read >= sizeof(flatbuffers::uoffset_t));
					flatbuffer_size = *(flatbuffers::uoffset_t*)(buffer);
				}

				total_bytes_read += current_bytes_read;
				i32 next_idx = arrlen(flatbuffer_data);
				arraddn(flatbuffer_data, current_bytes_read);
				memcpy(&flatbuffer_data[next_idx], buffer, current_bytes_read);
				++packets_read;	
			}
		}
		while (current_bytes_read == 0 || (flatbuffer_size && total_bytes_read < flatbuffer_size.value()));

		if (arrlen(flatbuffer_data) > 0)
		{
			printf("We've got some data! Data Length: %td Packets Read: %i\n", arrlen(flatbuffer_data), packets_read);
		}

		// Interpret Flatbuffer data
		auto* update = Blender::LiveLink::GetSizePrefixedUpdate(flatbuffer_data);
		assert(update);
		if (auto objects = update->objects())
		{
			for (i32 idx = 0; idx < objects->size(); ++idx)
			{
				auto object = objects->Get(idx);
				if (auto object_name = object->name())
				{
					printf("\tObject Name: %s\n", object_name->c_str());
				}

				int unique_id = object->unique_id();
				bool visibility = object->visibility();

				auto object_location = object->location();
				if (!object_location)
				{
					continue;
				}

				auto object_scale = object->scale();
				if (!object_scale)
				{
					continue;
				}

				auto object_rotation = object->rotation();
				if (!object_rotation)
				{
					continue;
				}

				HMM_Vec4 location = HMM_V4(
					object_location->x(), 
					object_location->y(), 
					object_location->z(), 
					1
				);
				HMM_Vec4 scale = HMM_V4(
					object_scale->x(), 
					object_scale->y(), 
					object_scale->z(), 
					0
				);
				HMM_Quat rotation = HMM_Q(
					object_rotation->x(), 
					object_rotation->y(), 
					object_rotation->z(), 
					object_rotation->w()
				);

				// Used to ensure we aren't reading from this object's data on the main thread 
				state.updating_object_uid = unique_id;

				if (state.objects.contains(unique_id))
				{	
					Object& existing_object = state.objects[unique_id];
					sg_destroy_buffer(existing_object.storage_buffer);
					if (existing_object.has_mesh)
					{
						sg_destroy_buffer(existing_object.mesh.index_buffer);
						sg_destroy_buffer(existing_object.mesh.vertex_buffer);
					}
				}

				Object game_object = make_object(
					unique_id, 
					visibility, 
					location, 
					scale, 
					rotation
				);

				// If our object has a mesh, set it up here
				if (auto object_mesh = object->mesh())
				{
					sbuffer(Vertex) vertices = nullptr;
					if (auto flatbuffer_vertices = object_mesh->vertices())
					{
						for (i32 vertex_idx = 0; vertex_idx < flatbuffer_vertices->size(); ++vertex_idx)
						{
							auto vertex = flatbuffer_vertices->Get(vertex_idx);
							auto vertex_position = vertex->position();
							auto vertex_normal = vertex->normal();

							Vertex new_vertex = {
								.position = {
									.X = vertex_position.x(),
									.Y = vertex_position.y(),
									.Z = vertex_position.z(),
									.W = vertex_position.w(),
								},
								.normal = {
									.X = vertex_normal.x(),
									.Y = vertex_normal.y(),
									.Z = vertex_normal.z(),
									.W = vertex_normal.w(),
								},
							};

							arrput(vertices, new_vertex);
						}
					}

					sbuffer(u32) indices = nullptr;
					if (auto flatbuffer_indices = object_mesh->indices())
					{
						for (i32 indices_idx = 0; indices_idx < flatbuffer_indices->size(); ++indices_idx)
						{
							u32 new_index = flatbuffer_indices->Get(indices_idx);
							arrput(indices, new_index);
						}
					}

					// Set Mesh Data on Game Object
					game_object.has_mesh = true;
					game_object.mesh = make_mesh(vertices, arrlen(vertices), indices, arrlen(indices));
				}

				state.objects[unique_id] = game_object;
				// clear atomic updating object uid
				state.updating_object_uid = -1;
			}

			state.blender_data_loaded = true;
		}
	}

	shutdown(connection_socket,2);
	shutdown(state.blender_socket,2);
}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With all of this setup, you can send updates to a running game by repeatedly calling the operator in Blender. 

**Handling Live Updates **<br/>

Getting a big, initial update is great to first set up our scene, but we also want to allow for smaller updates as things change in blender. For that we can hook into one of blender's callbacks:     

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Python default

# Callback when depsgraph has finished updating
def depsgraph_update_post_callback(scene, depsgraph):
    if not depsgraph_update_post_callback.enabled:
        return

    updated_objects = []
    deleted_object_uids = []

    # Determine if any objects were deleted
    current_objects = set(obj.session_uid for obj in scene.objects)

    # Track the objects at the last update
    if hasattr(depsgraph_update_post_callback, "previous_objects"):
        previous_objects = depsgraph_update_post_callback.previous_objects
        # Find the difference (deleted objects)
        deleted_object_uids = list(previous_objects - current_objects)
        if len(deleted_object_uids) > 0:
            print(f"Deleted Objects UIDs: {', '.join(map(str, deleted_object_uids))}")
    
    # Store the current object names for the next update
    depsgraph_update_post_callback.previous_objects = current_objects

    for update in depsgraph.updates:
        update_id = update.id
        if isinstance(update_id, bpy.types.Object):
            if update.is_updated_transform or update.is_updated_geometry:
                # appending update_id won't work, need to look up object in scene.objects
                updated_object = scene.objects[update_id.name]
                updated_objects.append(updated_object)

    if len(updated_objects) > 0: 
        # Temporarily disable depsgraph_update_post_callback to prevent infinite recursion
        depsgraph_update_post_callback.enabled = False 

		# Send our partial update
        live_link_connection.send_object_list(
            updated_objects = updated_objects, 
            deleted_object_uids = deleted_object_uids
        )

        # Re-Enable depsgraph_update_post_callback
        depsgraph_update_post_callback.enabled = True 

# Enable depsgraph_update_post_callback. 
# Will be disabled to prevent recursion when depsgraph_update_post_callback is doing its work
depsgraph_update_post_callback.enabled = True

#...

# Actually register blender's depsgraph update post callback
bpy.app.handlers.depsgraph_update_post.append(depsgraph_update_post_callback)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Now that the live-update system is working, we've got a reasonably interactive method of modifying our running game world from blender. If we wanted to save out a world for loading later, we'd need only save a flatbuffer of a full-update of our scene. In subsequenty posts I plan on outlining how we might send other data to our game such as lighting, physics-objects, and gameplay. Feel free to follow development at <https://github.com/ForestCSharp/blender_live_link>. 

<!-- Markdeep: --><style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style><script src="../../markdeep.min.js" charset="utf-8"></script><script>window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")</script>