Facial Animation
Loading the base mesh.
As with the expressions, we need to parse the 3D mesh.
This time we want to extract the information we need, and add some information.
We need to know a texture coordinate that maps to the vertex so we can get the expression data for this vertex.
We need a new vertex structure to hold this, so that's the first stage.
This time we want to extract the information we need, and add some information.
We need to know a texture coordinate that maps to the vertex so we can get the expression data for this vertex.
We need a new vertex structure to hold this, so that's the first stage.
struct MyOwnVertexFormat { private Vector3 position; private Vector2 texcoords; private Vector2 index; public MyOwnVertexFormat(Vector3 position, Vector2 texcoords, Vector2 index) { this.position = position; this.texcoords = texcoords; this.index=index; } public static VertexElement[] VertexElements = { new VertexElement(0, 0, VertexElementFormat.Vector3, VertexElementMethod.Default, VertexElementUsage.Position, 0), new VertexElement(0, sizeof(float)*3, VertexElementFormat.Vector2, VertexElementMethod.Default, VertexElementUsage.TextureCoordinate, 0), new VertexElement(0, sizeof(float)*5, VertexElementFormat.Vector2, VertexElementMethod.Default, VertexElementUsage.TextureCoordinate, 1), }; public static int SizeInBytes = sizeof(float) * (3 + 2 + 2); }
The only really important thing to notice is that we now have two texture coordinates. One for diffuse texture map, and one for the expression data.
Create a new class to hold your mesh I called mine MorphMesh, and add these member variables.
Create a new class to hold your mesh I called mine MorphMesh, and add these member variables.
private Model model; private MyOwnVertexFormat[] vertices; private byte[] oldverts; private VertexBuffer[] vbs; private VertexDeclaration vdec; private List<Expression>expressions= new List<Expression>(); private Dictionary<string, int> expmap = new Dictionary<string, int>();
Most of it is obvious. The dictionary is used when altering expressions, we will come to that later.
The list of expressions holds all the textures that define the expressions.
The first thing to do is load in the base mesh. Note we need at least one expression to have already been added to the MorphMesh before we can do this. This is caused by the need to know the size of the generated texture.
If we don't know the size of the texture, we cannot create the texture coordinates.
The list of expressions holds all the textures that define the expressions.
The first thing to do is load in the base mesh. Note we need at least one expression to have already been added to the MorphMesh before we can do this. This is caused by the need to know the size of the generated texture.
If we don't know the size of the texture, we cannot create the texture coordinates.
public void Load(GraphicsDevice graf, ContentManager cont, String file) { model = cont.Load<Model>(file); int dp = 0; vbs = new VertexBuffer[model.Meshes.Count]; for (int i = 0; i < model.Meshes.Count; i++) { // find the number of verts in this mesh int nverts = model.Meshes[i].VertexBuffer.SizeInBytes / model.Meshes[i].MeshParts[0].VertexStride; vertices = new MyOwnVertexFormat[nverts]; oldverts = new byte[model.Meshes[i].VertexBuffer.SizeInBytes]; // find the location of the position and texture data in the stream VertexElement [] vars = model.Meshes[i].MeshParts[0].VertexDeclaration.GetVertexElements(); int ppos = 0; int tpos = 0; for (int ve = 0; ve < vars.GetLength(0); ve++) { if (vars[ve].VertexElementUsage == VertexElementUsage.Position) { ppos = vars[ve].Offset; } if (vars[ve].VertexElementUsage == VertexElementUsage.TextureCoordinate) { tpos = vars[ve].Offset; } } model.Meshes[i].VertexBuffer.GetData<byte>(oldverts); // convert the vertex buffer int spos =0; for (int k = 0; k < nverts; k++) { float x = BitConverter.ToSingle(oldverts, spos + ppos); float y = BitConverter.ToSingle(oldverts, spos + ppos + 4); float z = BitConverter.ToSingle(oldverts, spos + ppos + 8); float tx = BitConverter.ToSingle(oldverts, spos + tpos); float ty = BitConverter.ToSingle(oldverts, spos + tpos + 4); int ex = dp % expressions[0].size; int ey = dp / expressions[0].size; float fex=(float)ex/(float)expressions[0].size; float fey=(float)ey/(float)expressions[0].size; vertices[k] = new MyOwnVertexFormat(new Vector3(x,y,z),new Vector2(tx,ty), new Vector2(fex,fey)); dp++; spos += model.Meshes[i].MeshParts[0].VertexStride; } vbs[i] = new VertexBuffer(graf,nverts*MyOwnVertexFormat.SizeInBytes,BufferUsage.WriteOnly); vbs[i].SetData<MyOwnVertexFormat>(vertices); } vdec = new VertexDeclaration(graf, MyOwnVertexFormat.VertexElements); }
The next method to add is the code to add an expression to the list. This is trivial.
public void AddExpression(GraphicsDevice graf, IServiceProvider service, String name, String mesh) { Expression exp = new Expression(); exp.Load(graf, service, mesh); expressions.Add(exp); expmap.Add(name, expressions.Count - 1); }
Now we need to be able to set the blend value for each expression.
I haven't added any error checking here, the result will be undefined if the sum of all blend factors is greater than one.
I haven't added any error checking here, the result will be undefined if the sum of all blend factors is greater than one.
public void SetExpressionWeight(string name, float value) { int offset = expmap[name]; expressions[offset].weight = value; }
Now we have the base mesh, we have the expressions, all that's left to do is draw it.
public void Draw(GraphicsDevice g, Effect shader) { int tex=0; foreach (Expression e in expressions) { string name = string.Format("expression_{0}", tex); shader.Parameters[name].SetValue(e.texture); name = string.Format("weight_{0}", tex); shader.Parameters[name].SetValue(e.weight); tex++; } shader.CommitChanges(); shader.Begin(); IEnumerator<ModelMesh> meshEnumerator = model.Meshes.GetEnumerator(); int i = 0; while (meshEnumerator.MoveNext()) { ModelMesh mesh = meshEnumerator.Current; g.Indices = mesh.IndexBuffer; IEnumerator<ModelMeshPart> meshPartEnumerator = mesh.MeshParts.GetEnumerator(); while (meshPartEnumerator.MoveNext()) { ModelMeshPart meshPart = meshPartEnumerator.Current; BasicEffect be = (BasicEffect)meshPart.Effect; shader.Parameters["diffuseTexture"].SetValue(be.Texture); g.VertexDeclaration = vdec; g.Vertices[0].SetSource(vbs[i], meshPart.StreamOffset, MyOwnVertexFormat.SizeInBytes); shader.CurrentTechnique.Passes[0].Begin(); g.DrawIndexedPrimitives ( PrimitiveType.TriangleList, meshPart.BaseVertex, 0, meshPart.NumVertices, meshPart.StartIndex, meshPart.PrimitiveCount ); shader.CurrentTechnique.Passes[0].End(); } i++; } shader.End(); }
As you can see, there is very little here. All the hard work is done in the vertex shader.
So I had better show you that hadn't I?
So I had better show you that hadn't I?
float4x4 World; float4x4 View; float4x4 Projection; //========================================================================================================// //= Textures //========================================================================================================// texture2D diffuseTexture; sampler diffuseSampler = sampler_state { texture = <diffuseTexture>; magfilter = LINEAR; minfilter = LINEAR; mipfilter = LINEAR; AddressU = WRAP; AddressV = WRAP; }; texture2D expression_0; float weight_0; sampler exp0_sampler = sampler_state { texture = <expression_0>; magfilter = POINT; minfilter = POINT; mipfilter = POINT; AddressU = CLAMP; AddressV = CLAMP; }; texture2D expression_1; float weight_1; sampler exp1_sampler = sampler_state { texture = <expression_1>; magfilter = POINT; minfilter = POINT; mipfilter = POINT; AddressU = CLAMP; AddressV = CLAMP; }; texture2D expression_2; float weight_2; sampler exp2_sampler = sampler_state { texture = <expression_2>; magfilter = POINT; minfilter = POINT; mipfilter = POINT; AddressU = CLAMP; AddressV = CLAMP; }; texture2D expression_3; float weight_3; sampler exp3_sampler = sampler_state { texture = <expression_3>; magfilter = POINT; minfilter = POINT; mipfilter = POINT; AddressU = CLAMP; AddressV = CLAMP; }; //========================================================================================================// //= Structures //========================================================================================================// struct VertexShaderInput { float4 Position : POSITION0; float2 TexCoords: TEXCOORD0; float2 Index : TEXCOORD1; }; struct VertexShaderOutput { float4 Position : POSITION0; float2 TexCoords: TEXCOORD0; }; VertexShaderOutput VertexShaderFunction(VertexShaderInput input) { VertexShaderOutput output; float4 spos = tex2Dlod(exp0_sampler,float4(input.Index,0,0)); float4 spos2 = tex2Dlod(exp1_sampler,float4(input.Index,0,0)); float4 spos3 = tex2Dlod(exp2_sampler,float4(input.Index,0,0)); float4 spos4 = tex2Dlod(exp3_sampler,float4(input.Index,0,0)); spos=lerp(input.Position,spos,weight_0); spos=lerp(spos,spos2,weight_1); spos=lerp(spos,spos3,weight_2); spos=lerp(spos,spos4,weight_3); float4 worldPosition = mul(spos, World); float4 viewPosition = mul(worldPosition, View); output.Position = mul(viewPosition, Projection); output.TexCoords=input.TexCoords; return output; } float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0 { float4 diffuseColor = tex2D(diffuseSampler, input.TexCoords); diffuseColor.a=1; return diffuseColor; } technique Technique1 { pass Pass1 { // TODO: set renderstates here. VertexShader = compile vs_3_0 VertexShaderFunction(); PixelShader = compile ps_3_0 PixelShaderFunction(); } }
Even that is not complex. The key bit to look at is the blending done in the vertex shader.
Be warned! The sampler states are very important here. Linear interpolation will screw everything up, and is not supported on floating point textures. So make sure you are using point sampling.
Thats it. All that I have left to do is add it to a XNA game and show you how to use it.
Be warned! The sampler states are very important here. Linear interpolation will screw everything up, and is not supported on floating point textures. So make sure you are using point sampling.
Thats it. All that I have left to do is add it to a XNA game and show you how to use it.