CPPN/ANNActivation Functions

Here is a collection of activation functions for Compositional Pattern Producing Networks, or CPPNs, which are a type of Artificial Neural Network, or ANN.

Activation functions vary by application, but most assume normalized inputs and outputs to the range of -1.0 and 1.0 (Bipolar Sigmoid, Hyperbolic Tangent) or between 0.0 and 1.0 (Gaussian). Other activation functions below do not assume normalized input (Sin, Cos, Tan, Step, Ramp, Spike, Inverse) and thus are useful for applications in which inputs are not restricted to the -1.0 to 1.0 range.

So why use CPPNs when theoretically, an ANN composed of only Sigmoids or Gaussians can potentially approximate any function? Because with multiple activation functions an ANN obtains variation more quickly and may possibly evolve toward a target sooner. CPPNs are useful for Interactive Evolutionary Computation (IEC) and Evolutionary Art applications where obtaining variety quickly and minimizing evolution time for users is a priority. Additionally, some of the activation functions presented are linear, enabling Evolutionary Art programs to easily generate straight lines, which is problematic for ANNs composed of strictly non-linear functions.

Note that in the code for sin(x), cos(x), tan(x), and tanh(x) you must #include "math.h" for C language or #include for C++. The graphs were generated with the CoolMath.com online graphing calculator.

Sine

Equation: sin(x)

Code: sin(x);

Cosine

Equation: cos(x)

Code: cos(x);

Tangent

Equation: tan(x)

Code: The discontinuity of tangent produces interesting results in NEAT Particles, therefore it may be useful in other Evolutionary Art applications. Since the standard C/C++ tan() function can return NAN (not a number), +INFINITY, -INFINITY, or a number, you must handle each case. I arbitrarily chose 10000.0f as a large number to return.

float tangent(float x)
{
    x = tan(x);
    if(x<10000.0f && x>-10000.0f) return x;
    if(x > 10000.0f) return 10000.0f;
    if(x < -10000.0f) return -10000.0f;
    return 0.0f;
}

Hyperbolic Tangent

Equation: sinh(x)/cosh(x) =

Code: tanh(x);

Bipolar Sigmoid

Equation:

Code:

float bipolarSigmoid(float x)
{
    return (1.0f-exp(-x)) / (1.0f+exp(-x));
}

Gaussian

Equation:

Code: You can greatly simplify the Gaussian formula as below if you don't mind that it approaches 1.0 at zero unlike the classic Gaussian function.

float gaussian(float x)
{
    return exp(-1.0f * (x*x));
}

Ramp

Equation: 1 - 2*(x-floor(x))

Code:

float ramp(float x)
{
    return 1.0f - 2.0f * (x-floor(x));
}


Step

Equation: if(floor(x) is even) 1.0 else -1.0

Code:

float step(float x)
{
    if((int)floor(x)%2 == 0.0f) return 1.0f;
    else return -1.0f;
}

Spike

Equation: if(floor(x) is even) 1 - 2*(x-floor(x)) else -1 + 2*(x-floor(x))

Code: Basically a pointy version of sin or cos.

float spike(float x)
{
    if((int)floor(x)%2 == 0) return 1.0f - 2.0f * (x-floor(x));
    else return -1.0f + 2.0f * (x-floor(x));
}

Inverse

Equation: -x

Code: Since inverse flips the function in the preceeding node it may be useful for producing mirror images or tranforms.

float inverse(float x)
{
    return -x;
}