Activation Functions¶

Choosing a suitable activation function for the task at hand can make a significant difference, depending on the situation or dataset. In order to give you a certain freedom in this sense, Fortnet implements the following functions:

hyperbolic tangent

sigmoid function

gaussian function

relu function

heaviside function

linear function

The activation function is selected in the Network block of the HSD input. The functions and an associated, exemplary network block, are listed below.

Note

A linear function is always used as the activation function of the output layer, in order to be able to represent all real-valued results.

Hyperbolic Tangent¶

Network = BPNN {
  Hidden = 2 2
  Activation = tanh
}

Sigmoid¶

Network = BPNN {
  Hidden = 2 2
  Activation = sigmoid
}

Gaussian¶

Network = BPNN {
  Hidden = 2 2
  Activation = gaussian
}

ReLU¶

Network = BPNN {
  Hidden = 2 2
  Activation = relu
}

Heaviside¶

Network = BPNN {
  Hidden = 2 2
  Activation = heaviside
}

Linear¶

Network = BPNN {
  Hidden = 2 2
  Activation = linear
}