Indico avec slides: https://indico.in2p3.fr/event/32012/

Presentation by Michael Dell’aiera and Antoine Bralet

Notes

presentation of different norms by Antoine and Michaël

Normalization layers (Michael)

https://wandb.ai/gammalearn/vanilla_norm/reports/Normalization-layers--Vmlldzo2ODY3MjMy?accessToken=20m2ett6lrxz99oya85k375gqrmuhsuj3hb498h9a788nehk72txh9hf2yz7tqyx

• Introduction to Normalization Layers:
  • In a previous era, training time was very large, we needed a method to reduce it.
  • One tip was about normalization to improve convergence.
• What it allow:
  • Accelerate training
  • Solve vanishing gradients
  • Reduce the effect of parameter initialization
• Batch norm is used a lot:
  • It is applied on the batch axis.
  • If the batch size is very small, you could get biased statistics using it.
        -You can use it in many different configurations within the network.
• Michaël showed some results:
  • Experiments from Michael on Monte-Carlo data for gamma-ray astronomy and different norms.
  • With different configurations, some give better/more stable results than others
  • Strangely enough, the configuration with no batch norm showed the better results in the classification task.
  • The other results showed increased performances when using batchnorm.

Advantages of other norms:

• when you can’t compute batch norm, e.g. for RNNs
• small batches: batch norm on a small batch has less meaning

Antoine presents conditionned norm

context: multi-input networks with major input(s) and additional input(s) that help the decision

• When you have a “main input” and “additional input”, you need to incorporate these data in a fancy way.
• The conditioned batch normalization have two sets of parameters:
  • One is calculated like the regular batch normalization case.
  • The other is determined by the encoded “additional part” and passed through dense layers to predict the other two parameters of the conditioned batch normalization layer.
  • Apparently this shows great results and looks like a promising opportunity in some applications.

Mailing list

clubdata.idefics@services.cnrs.fr

To subscribe:

Send a message to sympa@services.cnrs.fr from the address you want to subscribe to the list.

Sympa is not a person but a mailing list management robot. Thus it is useless to send it loving words! ;-)

In the subject line of your message, type in: subscribe clubdata.idefics Firstname Name (indicate your own first name and name).
Leave the message body blank.

Next meeting

Kavé ?
19/03/2024