Assets

Created	@Jun 16, 2019 10:47 AM
Tags

Overview

A game contains a very large number of assets.

There are two category of assets.

External assets produced by a third-party application (mesh, texture, ...)

Internal assets produced in the editor (scenes, entities, materials, settings, scripts, ...)

Assets have different form: texture, shader, material, script, ...

Assets need some way of being referenced.

External assets: a mesh references materials which in turn reference shaders and textures.

Internal assets: an entity spawns other entities from a specific location.

These references need to be resolved at runtime.

Asset Names

There are three ways to identify assets:

Name

Identifier

Combination

Names

Names reference assets by their role.

File Path

A name is a string associated with an asset that can be changed by the user.

Identifying external assets by name is the easiest method as the external assets are imported from the operating system's file system which is already based on file names.

Models/Buildings/Hotel.fbx

The file names should have a canonical path:

Only use relative paths

Force to lower case

Only use foward slashes only as directory separator

Do not allow dots

Display Name

Internal assets can have a generated name (based on their type or context of creation).

The user can then rename the assets at will.

In the case of hierarchical assets, they can be identified with a unique assets path.

NewScene/Environment/Trees/PalmTree[2]

✔Easy for users to identify assets.

✔Easy to implement.

✔Late binding to assets. A referenced assets can be replaced by a new asset without updating the references.

❌Moving or renaming assets breaks references.

❌Tedious to define a unique name for each asset.

❌String comparisons are expensive.

❌Generated names are confusing (ie. Box842).

Identifiers

Unique Identifier

Unique identifiers reference assets by their identity.

A unique identifier is associated with an asset when it is created, and is never changed.

There are two methods to associated a unique identifier with an asset:

The asset can be stored on disk using its identifier instead of a regular file name.

The asset can have additional metadata in which its identifier is stored.

RFC 4122 UUID: 536bd1f9-699a-4bc8-b046-6224ea901d21

Hash

Hashes reference assets by their content.

An alternative is to generate a hash from the content of an asset.

In that case, the assets is referenced by name at edit-time, and then using the computed hash at run-time.

MD5: 50481dd8a9afdbfbb768378acb5c1c6b

❌Difficult for users to identify assets.

❌Impossible to work with third-party applications.

Combination

A better solution is to associate a unique identifier with assets but work with assets using a display name during development which is stripped from the final game.

Renaming assets is now a simple operation as the editor can fix the references automatically.

A reference to an asset would look like this.

texture:
{
  path: "textures/my_texture"
  id: "5e1b607e-6fea-4e03-9946-15395e9c305d"
}

Asset Identifiers

There are multiple method to compute unique identifiers for assets.

A naive approch can create collisions: multiple strings with an identical hash value.

Some are based on hash values. A hash value is a number calculated from the bits of the string using some algorithm to produce a fixed number of bits.

Many common algorithms produce large numbers (more than 128-bits) such as SHA, MD5, ...

Some algorithms produce 32-bit or 64-bit values but cansuch as: CRC-32, FNV-1a, ...

FNV-1a hash

A FNV-1a hash can be computed from the asset name and an offset and prime values.

The offset and prime values can be compile-time constants.

The algorithms can be implemented as a 32-bit or 64-bit hash function.

  static size_t CalculateFNV(const char* str, size_t offset, size_t prime)
  {
    size_t hash = offset;
    while (*str != 0)
    {
      hash ^= *str++;
      hash *= prime;
    }
 
    return hash;
  }

Assets and code

To be able to use assets in code, there are multiple methods.

Look-ups

Strings

The most simple case is to look-up a resource based on a string, that represents its path or its name.

Texture* asset = TextureManager::FindTexture("my_texture");

Or in a more general way.

Asset* asset = AssetManager::Find("my_texture");

Whenever we have to find a certain resource based on a name, we could do one of the following:

Store each resource’s name, and compare it with the name of the requested resource.

Compute and store the hash of each resource’s name, and compare it with the hash of the requested resource’s name.

To compare asset's name, we can just perform a string comparison:

Asset* AssetManager::Find(const char* name)
{
  for (auto asset : _assets)
    if (strcmp(asset->GetName(), name) == 0)
      return asset;
  return nullptr;
}

There are a few problems with this method:

❌When an asset is renamed, the code still compiles but fails at runtime.

❌String take up a large amount of memory (especially with paths).

❌String comparisons are very slow.

❌Cache trashing occurs because string have a variable-length.

Hash

Comparing hashes would be more performant and could initially be implemented as such:

Asset* AssetManager::Find(const char* name)
{
  size_t hash = CalculateHash(name);

  for (auto asset : _assets)
    if (asset->GetHash() == hash)
      return asset;
  return nullptr;
}

❌The problem with this method is that in many cases, the name parameter will be constant in the game code, but the hash will be computer with every request.

Hash values

A basic solution is to look-up resources with hashes directly.

Texture* asset = AssetManager::Find<Texture>(0x1234abcd);

✔No runtime cost.

❌Easy to make mistakes.

❌The code is hard to understand. Adding a comment is poor improvement as comments can be out of sync.

Macros

An improvement over this method is to generate a header file with macros of all the assets in the game.

#pragma once

#define ASSET_MYTEXTURE    0x1234abcd
#define ASSET_OTHERTEXTURE 0x5678cdef

Instead of using the hash value directly, we can include the generated header and use the macro.

Texture* asset = AssetManager::Find<Texture>(ASSET_MYTEXTURE);

✔When an asset is renamed the macro changes, and the compilation fails.

❌The header file has to be included everywhere a hash value is needed.

❌The header file has to be updated when new assets are added and many files need to be recompiled.

Static hashes

Instead, we can performs lookups using both the file name and its precomputed static hash inplace.

Texture* asset = AssetManager::Find<Texture>(StaticHash("my_texture", 0x1234abcd));

The StaticHash() function computes the hash value in debug and asserts if the computed value is different from the static hash value, to ensure that the code is safe.

In release, the StaticHash() function ignores the file name which is then stripped from the code.

#ifdef DEBUG
  inline size_t StaticHash(const char* name, size_t value)
  {
    assert(CalculateHash(name) == value);
    return value;
  }
#else
  #define StaticHash(str, value) (value)
#end

The pitfall of this technique is that if the asset is renamed, the omputed hash will be valid but the asset won't be found.

Hash strings

Instead we can implement a HashString data structure that will compute the hash once for literal strings and store it.

We define two constructors:

A constructor that takes constant strings and compute a hash value every time.

A constructor that only takes static strings and look-up the corresponding hash from the address of the string; otherwise only compute and store the hash value once.

struct HashString
{
private:
  static Set<const char*, size_t> _hashes;
  size_t _hash;

public:
  HashString(const char* str)
  {
    _hash = ComputeHash(str);
  }

  template<int N>
  HashString(char (&str)[N])
  {
    _hash = str.Get(str);
    if (!IsValidHash(_hash))
    {
      _hash = ComputeHash(str);
      _hashes[str] = _hash;     
    }
  }

  size_t Get() const { return _hash; }
};

This code will failed because the constuctor that takes a const char* will never be called.

We can fix it by defining a ConstChar wrapper structure on a const char*.

struct ConstChar
{
private:
  const char* _str;

public:
  ConstChar(const char* str) : _str(str) {}

  const char* Get() const { return _str; }
};

We can then modify the HashString constructor to use a ConstChar instead of a const char*.

struct HashString
{
  ...

  HashString(ConstChar str)
  {
    _hash = ComputeHash(str);
  }

  ...
};

The look-up function can now compare the hash values, and store the assets into a set for better performances.

Asset* AssetManager::Find(HashString name)
{
  return _assets[name.Get()];
}

Static assert

The solution can be improved further by making the hash function and the StaticHash function constexpr functions. The assert can then be run at compile-time using static_assert.

#ifdef DEBUG
  constexpr size_t StaticHash(const char* name, size_t value)
  {
    static_assert(CalculateHash(name) == value);
    return value;
  }
#else
  #define StaticHash(str, value) (value)
#end

Postbuild step

A different solution from the static hash method is to use file names wrapped into a macro.

The macro is defined as an empty macro, but a postbuild step parses all the source code to replace its occurences with the calculated hash values.

#define HASH()

Texture* asset = AssetManager::Find<Texture>(HASH("my_texture"));

✔No runtime cost.

❌Complexify the build process.

Asset references

Using asset references with the editor can circumvent the problem.

Every asset reference is stored as a unique identifier but displayed in the editor using its display name.

When the game code needs an asset, it declares a serializable asset reference in its data structure that is set in the editor.

When the game loads, the reference is deserialized and resolved. The game code just needs to check for invalid references, but doesn't need to explicitly load the asset in code.