unreal.NearestNeighborModel¶
- class unreal.NearestNeighborModel(outer: Object | None = None, name: Name | str = 'None')¶
Bases:
MLDeformerMorphModelThe nearest neighbor model. This model contains the linear basis of the vertex deltas and a small set of meshes for nearest neighbor search. Given a new pose, the pre-trained neural network first predicts the coefficients of the vertex deltas. Then this model uses the predicted coeffcients to find a nearest neighbor in the small dataset. The total vertex delta is computed by
vertex_delta = mean_delta + basis * coeff + nearest_neighbor_delta
- To prevent popping, a time filtering is applied on predicted vertex deltas. The vertex delta at time t is computed by
vertex_delta(t) = decay_factor * vertex_delta(t-1) + (1 - decay_factor) * vertex_delta
The mesh can be separated into several sections (e.g. shirt, pants…). The nearest neighbor search is carried out separately for each section. The basis and the nearest neighbor data are compressed into morph targets.
C++ Source:
Plugin: NearestNeighborModel
Module: NearestNeighborModel
File: NearestNeighborModel.h
Editor Properties: (see get_editor_property/set_editor_property)
alignment_transform(Transform): [Read-Write] The transform that aligns the Geometry Cache to the SkeletalMesh. This will mostly apply some scale and a rotation, but no translation.anim_sequence(AnimSequence): [Read-Write] The animation sequence to apply to the base mesh. This has to match the animation of the target mesh’s geometry cache. Internally we force the Interpolation property for this motion to be “Step”. deprecated: Use the training input anims instead.batch_size(int32): [Read-Write] Number of data samples processed together as a group in a single pass.clamp_morph_weights(bool): [Read-Write] Should we enable morph target weight clamping? The minimum and maximum values that it will be clamped against will be the min/max morph target weight values that have been seen when running the training dataset through the network. The advantage of clamping is that it can make deformations more stable when we have input poses that have not been seen during training. We basically prevent the weights from ‘exploding’ and getting very large values, which could make the mesh look very bad.decay_factor(float): [Read-Write] The ratio of previous frame deltas added into the current frame deltas. A bigger value will make wrinkles “stick” longer.delta_cutoff_length(float): [Read-Write] Sometimes there can be some vertices that cause some issues that cause deltas to be very long. We can ignore these deltas by setting a cutoff value. Deltas that are longer than the cutoff value (in units), will be ignored and set to zero length.early_stop_epochs(int32): [Read-Write] The number of epochs to stop training if there is no improvement in accuracy.file_cache_directory(str): [Read-Only] Directory to save the intermediate results.geometry_cache(GeometryCache): [Read-Write] The geometry cache that represents the target deformations. deprecated: Use the training input anims instead.hidden_layer_dims(Array[int32]): [Read-Write] Dimension of hidden layers in the network. This cannot be empty.include_normals(bool): [Read-Write] Include vertex normals in the morph targets? The advantage of this can be that it is higher performance than recomputing the normals. The disadvantage is it can result in lower quality and uses more memory for the stored morph targets.input_dim(int32): [Read-Only] Network input dimensions.input_multipliers(Array[Vector3f]): [Read-Write] Values to be multiplied to the input. This can be used to debug bad network input.inputs_max(Array[float]): [Read-Write] The max input values observed throughout the entire training set. This is used to clamp the input value at inference time. The values are set in python.inputs_min(Array[float]): [Read-Write] The min input values observed throughout the entire training set. This is used to clamp the input value at inference time. The values are set in python.invert_mask_channel(bool): [Read-Write] Enable this if you want to invert the mask channel values. For example if you painted the neck seam vertices in red, and you wish the vertices that got painted to NOT move, you have to invert the mask. On default you paint areas where the deformer should be active. If you enable the invert option, you paint areas where the deformer will not be active.learning_rate(float): [Read-Write] The size of the step when optimizing the network.mask_channel(MLDeformerMaskChannel): [Read-Write] The channel data that represents the delta mask multipliers. You can use this feather out influence of the ML Deformer in specific areas, such as neck line seams, where the head mesh connects with the body. The painted vertex color values will be like a weight multiplier on the ML deformer deltas applied to that vertex. You can invert the mask as well.max_num_lo_ds(int32): [Read-Write] How many Skeletal Mesh LOD levels should we generate MLD lods for at most? Some examples: A value of 1 means we only store one LOD, which means LOD0. A value of 2 means we support this ML Deformer on LOD0 and LOD1. A value of 3 means we support this ML Deformer on LOD0 and LOD1 and LOD2. We never generate more LOD levels for the ML Deformer than number of LOD levels in the Skeletal Mesh, so if this value is set to 100, while the Skeletal Mesh has only 4 LOD levels, we will only generate and store 4 ML Deformer LODs. The default value of 1 means we do not support this ML Deformer at LOD levels other than LOD0. When cooking, the console variable “sg.MLDeformer.MaxLODLevelsOnCook” can be used to set the maximum value per device or platform.max_training_frames(int32): [Read-Write] The maximum numer of training frames (samples) to train on. Use this to train on a sub-section of your full training data.morph_compression_level(float): [Read-Write] The morph target compression level. Higher values result in larger compression, but could result in visual artifacts. Most of the times this is a value between 20 and 200.morph_delta_zero_threshold(float): [Read-Write] Morph target delta values that are smaller than or equal to this threshold will be zeroed out. This essentially removes small deltas from morph targets, which will lower the memory usage at runtime, however when set too high it can also introduce visual artifacts. A value of 0 will result in the highest quality morph targets, at the cost of higher runtime memory usage.nearest_neighbor_offset_weight(float): [Read-Write] The weight multiplied to nearest neighbor deltas. A value of 0 means completely removing nearest neighbor deltas.num_basis_per_section(int32): [Read-Write] The number of basis used in each section. Only editable when UsePCA is false.num_epochs(int32): [Read-Write] Max number of cycles iterated through the training set.output_dim(int32): [Read-Only] Network output dimensions.rbf_sigma(float): [Read-Write] Range to blend nearest neighbors. A bigger range will blend more neighbors, produce smoother result and may be slower in game.regularization_factor(float): [Read-Write] The regularization factor. Higher values can help generate more sparse morph targets, but can also lead to visual artifacts.sections(Array[NearestNeighborModelSection]): [Read-Write]skeletal_mesh(SkeletalMesh): [Read-Write] The skeletal mesh that represents the linear skinned mesh.smooth_loss_beta(float): [Read-Write] The beta parameter in the smooth L1 loss function, which describes below which absolute error to use a squared term. If the error is above or equal to this beta value, it will use the L1 loss.training_input_anims(Array[MLDeformerGeomCacheTrainingInputAnim]): [Read-Write]use_dual_quaternion_deltas(bool): [Read-Write] Whether to use dual quaternion deltas. If false, LBS deltas will be used.use_file_cache(bool): [Read-Write] Whether to cache intermediate results on disk. CAUTION: failing to manually clear caches could cause unexpected results.use_input_multipliers(bool): [Read-Write] Whether to use input multipliers. This can be used to debug bad network input.use_pca(bool): [Read-Write] Whether to use pre-computed PCA basis. If false, basis will be learned at training time.use_rbf(bool): [Read-Write] Whether to use radial basis function to blend multiple nearest neighbors to produce smoother result.
- property batch_size: int¶
[Read-Write] Number of data samples processed together as a group in a single pass.
- Type:
(int32)
- property decay_factor: float¶
[Read-Write] The ratio of previous frame deltas added into the current frame deltas. A bigger value will make wrinkles “stick” longer.
- Type:
(float)
- property early_stop_epochs: int¶
[Read-Write] The number of epochs to stop training if there is no improvement in accuracy.
- Type:
(int32)
- property file_cache_directory: str¶
[Read-Only] Directory to save the intermediate results.
- Type:
(str)
- get_batch_size() int32¶
Get Batch Size
- Return type:
int32
- get_early_stop_epochs() int32¶
Get Early Stop Epochs
- Return type:
int32
Get Hidden Layer Dims
- Return type:
Array[int32]
- get_input_dim() int32¶
Get Input Dim
- Return type:
int32
- get_num_epochs() int32¶
Get Num Epochs
- Return type:
int32
- get_num_sections() int32¶
~END UMLDeformerModel overrides.
- Return type:
int32
- get_output_dim() int32¶
Get Output Dim
- Return type:
int32
- get_section_ptr(index) NearestNeighborModelSection¶
GetSectionPtr is reserved for python. Use GetSection for C++.
- Parameters:
index (int32) –
- Return type:
- get_total_num_basis() int32¶
Get Total Num Basis
- Return type:
int32
[Read-Write] Dimension of hidden layers in the network. This cannot be empty.
- Type:
(Array[int32])
- property input_multipliers: None¶
[Read-Write] Values to be multiplied to the input. This can be used to debug bad network input.
- property inputs_max: None¶
[Read-Write] The max input values observed throughout the entire training set. This is used to clamp the input value at inference time. The values are set in python.
- property inputs_min: None¶
[Read-Write] The min input values observed throughout the entire training set. This is used to clamp the input value at inference time. The values are set in python.
- property learning_rate: float¶
[Read-Write] The size of the step when optimizing the network.
- Type:
(float)
- property nearest_neighbor_offset_weight: float¶
[Read-Write] The weight multiplied to nearest neighbor deltas. A value of 0 means completely removing nearest neighbor deltas.
- Type:
(float)
- property num_basis_per_section: int¶
[Read-Write] The number of basis used in each section. Only editable when UsePCA is false.
- Type:
(int32)
- property num_epochs: int¶
[Read-Write] Max number of cycles iterated through the training set.
- Type:
(int32)
- property rbf_sigma: float¶
[Read-Write] Range to blend nearest neighbors. A bigger range will blend more neighbors, produce smoother result and may be slower in game.
- Type:
(float)
- property regularization_factor: float¶
[Read-Write] The regularization factor. Higher values can help generate more sparse morph targets, but can also lead to visual artifacts.
- Type:
(float)
- property smooth_loss_beta: float¶
[Read-Write] The beta parameter in the smooth L1 loss function, which describes below which absolute error to use a squared term. If the error is above or equal to this beta value, it will use the L1 loss.
- Type:
(float)
- property use_dual_quaternion_deltas: bool¶
[Read-Write] Whether to use dual quaternion deltas. If false, LBS deltas will be used.
- Type:
(bool)
- property use_file_cache: bool¶
failing to manually clear caches could cause unexpected results.
- Type:
(bool)
- Type:
[Read-Write] Whether to cache intermediate results on disk. CAUTION
- property use_input_multipliers: bool¶
[Read-Write] Whether to use input multipliers. This can be used to debug bad network input.
- Type:
(bool)