Bye to Trial and Error Activation Functions of Neural Networks II: Proposed Jameel’s ANNAF…

Source: Deep Learning on Medium

Bye to Trial and Error Activation Functions of Neural Networks II: Proposed Jameel’s ANNAF Stochastic Criterion

Artist Hans Hoffman wrote, The ability to simplify means to eliminate the unnecessary so that the necessary may speak.”

Casper Hansen (2019) says Better optimized neural network; choose the right activation function, and your neural network can perform vastly better.

In my second article entitled “Scientific Facts that prove the existing set of Activation Functions formed the Neural Networks’ Black-Box”, we stopped at a point where we are about to introduce the proposed “Jameel’s ANNAF Stochastic Criterion (2019)”.

Jamilu (2019) proposed “JAMEEL’S ANNAF STOCHASTIC CRITERION” as follows. Note that ANNAF means Artificial Neural Network Activation Functions.

This stated that under this criterion, we run the goodness of fits test on our referenced PURIFIED AL-ML-DATA SET such that:

(1) We accept if the Average of the ranks of Kolmogorov Smirnor, Anderson Darling and Chi-squared is less than or equal to Three (3);

(2) We must choose the fat-tailed Probability Distribution follows by our referenced PURIFIED AL-ML-DATA SET ITSELF regardless of its Rankings;

(3) If there is a tie, we include both the fat-tailed Probability Distributions in the selection;

(4) At least two (2) fat-tailed Probability Distributions must be included in the selection;

(5) We select the most occur Probability Distribution as the qualified candidate in each case of a test of goodness of fit on our referenced PURIFIED AL-ML-DATA SET;

More so, Jamilu (2019) proposed “Jameel’s Stochastic Lemma” as follows:

All the TOP-RANKED Fat-tailed Monotone Continuously Differentiable Stochastic Functions EMANATED from referenced AI-ML-Purified Data satisfies Proposed “Jameel’s Stochastic ANNAF Criterion” are EXCELLENT STOCHASTIC ACTIVATION FUNCTIONS to perform well-informed Forward and Backward Propagations of a Deep Learning Neural Network.

In July 2015, the author considered STOCKS PRICES of Eleven (11) out of Fifty (50) World’s Biggest Public Companies by FORBES as of 2015, applied “Jameel’s Criterion (first version)” to came up with the following Stochastic Functions which he subsequently proposed in October, 2019 are INDEED EXCELLENT set of STOCKS PRICES ACTIVATION FUNCTIONS to conducts STOCKS PRICES DEEP LEARNING NEURAL NETWORKS. Thus, the RANKING of the Stocks Prices Advanced Optimized Activation Functions are as follows:

First Derivative of Log — Logistic (3P) Probability Distribution (1st):

First Derivative of Pearson 5 (3P) Probability Distribution (3rd):

First Derivative of Burr (4P) Probability Distribution (4th):

First Derivative of Fatique Life (3P) Probability Distribution (5th):

Here, you will apply Product and Chain Rules. The following gives the Derivative of a Normal Distribution, this idea can be applied to solve the above problem.

First Derivative of Dagum (4P) Probability Distribution (7th):

First Derivative of Lognormal (3P) Probability Distribution (8th):

NOTE: The First Derivatives’ of the above EIGHT(8) Activation Functions’ LIMITS (RANGES) are not stated in this article.

Note that the author did not check the Monotone Differentiability of the functions presented above. However, the market trends are very volatile, many things had happened from 2019–2015. The distribution time series Data was from 2014–1990, was shown in (2015) showed. Different results may be obtained when conducted in the year 2019. Anyway, the author adopted the fat-tailed probability distributions obtained as per as 2015.

This research work was published in the “Risk and Financial Management Journal, IDEASPREAD.ORG, USA” and available online: https://j.ideasspread.org/index.php/rfm/article/view/387

To be continued in the next article.

AUTHOR

Jamilu Auwalu Adamu, FIMC, CMC, FIMS (UK), FICA (in view)

Associate Editor, Risk and Financial Management Journal, USA

Editor, Journal of Economics and Management Sciences, USA

Former Associate Editor, Journal of Risk Model Validation, UK

PEER-REVIEWER, RISK.NET Journals, London

Former, Steering Committee Member, PRMIA Nigeria Chapter

Books Author

Correspondence: Mathematics Programme Building, 118 National Mathematical Centre, Small Sheda, Kwali, 904105, FCT-Abuja, Nigeria. Tel: +2348038679094. E-mail: whitehorseconsult@yahoo.com

References

TABLECURVE 2D SOFTWARE, SYSTAT (2019) available online: https://systatsoftware.com/products/tablecurve-2d/tablecurve-2d-curve-fitting/

Derivative Calculator (2019) available on: https://www.derivative-calculator.net/

Jamilu Auwalu Adamu (2019), Advanced Stochastic Optimization Algorithm for Deep Learning Artificial Neural Networks in Banking and Finance Industries, Risk and Financial Management Journal, USA, Vol 1 No1 (2019), DOI: https://doi.org/10.30560/rfm.v1n1p8 and available online: https://j.ideasspread.org/index.php/rfm/article/view/387

Jamilu Auwalu Adamu (2019), Superintelligent Deep Learning Artificial Neural Networks, accepted for publication in the International Journal of Applied Science, IDEAS SPREAD. INC, USA (https://j.ideasspread.org/index.php/ijas), preprint available on https://www.preprints.org/manuscript/201912.0263/v1 with doi: 10.20944/preprints201912.0263.v1

Jamilu Auwalu Adamu (2019), Advanced Deterministic Optimization Algorithm for Deep Learning Artificial Neural Networks, accepted for publication in the International Journal of Applied Science, IDEAS SPREAD. INC, USA (https://j.ideasspread.org/index.php/ijas)

Jamilu Auwalu Adamu (2019), Deterministic and Stochastic Superintelligent Digital Brains, Distinct Biological Neurons: Distinct Activation Functions implying Distinct Artificial Neurons, preprints available on researchgate.net and academia.edu respectively via https://www.researchgate.net/publication/338170126_Deterministic_and_Stochastic_Superintelligent_Digital_Brains_Distinct_Biological_Neurons_Distinct_Activation_Functions_implying_Distinct_Artificial_Neurons with DOI: 10.13140/RG.2.2.31550.23368 and https://www.academia.edu/41430249/Deterministic_and_Stochastic_Superintelligent_Digital_Brains

Jamilu Auwalu Adamu (2019), Backward Propagation of Artificial Neural Network: First Derivatives of Advanced Optimized Stochastic Activation Functions, preprints available on researchgate.net via https://www.researchgate.net/publication/337907110_Backward_Propagation_of_Artificial_Neural_Network_First_Derivatives_of_Advanced_Optimized_Stochastic_Activation_Functions , DOI: 10.13140/RG.2.2.14004.60803

Jamilu Auwalu Adamu (2015),Banking and Economic Advanced Stressed Probability of Default Models, Asian Journal of Management Sciences, 03(08), 2015, 10–18.

Jamilu A. Adamu (2015), Estimation of Probability of Default using Advanced Stressed Probability of Default Models, Ongoing Ph.D Thesis, Ahmadu Bello University (ABU), Zaria, Nigeria.

Nair et al. (2010), Rectified linear units improve restricted boltzmann machines, ICML’10 Proceedings of the 27th International Conference on International Conference on Machine Learning Pages 807–814, Haifa, Israel — June 21–24, 2010.

Djork-Arne Clevert, Thomas Unterthiner & Sepp Hochreiter (2016), FAST AND ACCURATE DEEP NETWORK LEARNING BY EXPONENTIAL LINEAR UNITS (ELUS), Published as a conference paper at ICLR 2016

Klambauer et al. (2017),Self-Normalizing Neural Networks, Institute of Bioinformatics, Johannes Kepler University Linz, Austria.

Lichman, 2013, UCI machine learning repository, URL http://archive. ics. uci. edu/ml 901

Aman Dureja and Payal Pahwa (2019), Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks, Recent Patents on Computer Science Journal, Volume 12 , Issue 3 , 2019, DOI : 10.2174/2213275911666181025143029

Chigozie Enyinna Nwankpa et al. (2018), Activation Functions: Comparison of Trends in Practice and Research for Deep Learning, available online: https://arxiv.org/pdf/1811.03378.pdf

Soufiane Hayou et al. (2019), On the Impact of the Activation Function on Deep Neural Networks Training, available online: https://arxiv.org/pdf/1902.06853.pdf

Schoenholz et al. (2017), DEEP NEURAL NETWORKS AS GAUSSIAN PROCESSES, Published as a conference paper at ICLR 2018, available online: file:///C:/Users/pc/Downloads/Deep_Neural_Networks_as_Gaussian_Processes.pdf

Asman Dureja and Payal Pahwa (2019), Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks, Recent Patents on Computer Science, Volume 12 , Issue 3 , 2019, DOI : 10.2174/2213275911666181025143029

Casper Hansen (2019) says “Better optimized neural network; choose the right activation function, and your neural network can perform vastly better”, available online: https://mlfromscratch.com/neural-networks-explained/#/

Artist Hans Hoffman wrote, “The ability to simplify means to eliminate the unnecessary so that the necessary may speak.” available online: https://www.brainyquote.com/quotes/hans_hofmann_107805

Barnaby Black et.at (2016), Complying with IFRS 9 Impairment Calculations for Retail Portfolios, Moody’s Analytics Risk Perspectives, the convergence of Risk, Finance, and Accounting, Volume VII, June, 2016.

Ben Steiner (2019), Model Risk Management for Deep Learning and Alpha Strategies, BNP Paribas Asset Management, Quant Summit 2019

Bellotti T. and Crook J. (2012), Loss Given Default Models Incorporating Macroeconomic Variables for Credit Cards, International Journal of Forecasting, 28(1), 171–182, DOI: 10.1016/j.ijforecast.2010.08.005

Burton G. Malkiel (2009), The Clustering of Extreme Movements: Stock prices and the Weather, Princeton University, AtanuSaha, Alixpartners, Alex Grecu, Huron Consulting Group, CEPS working paper №186 February, 2009.

Chigozie Enyinna Nwankpa et al. (2018), Activation Functions: Comparison of Trends in Practice and Research for Deep Learning, Preprint.

Nassim N. Taleb (2011), The Future has Thicker Tails Past: Model Error as Branching Counterfactuals, presented in Honor of Benoit Mandelbrot’s at his Scientific Memorial, Yale University, April, 2011.

Nassim N. Taleb (2011), A Map and Simple Heuristic to Detect Fragility, Antifragility, and Model Error, First Version, 2011

Nassim N. Taleb (2010), Why Did the Crisis of 2008 Happen, Draft, 3rd Version, August, 2010.

Nassim N. Taleb (2009), Errors, Robustness, and Fourth Quadrant, New York University Polytechnic Institute and Universa Investment, United States, International Journal of Forecasting 25 (2009) 744–759

Nassim N. Taleb (2010), Convexity, Robustness, and Model Error inside the “Black Swan Domain”, Draft Version, September, 2010.

Nassim N. Taleb et al (2009), Risk Externalities and Too bid to Fail, New York University Polytechnic Institute, 11201, New York, United States.

Nassim N. Taleb (2012), The Illusion of Thin — Tails under Aggregation, NYU — Poly, January, 2012

Nassim N. Taleb (2007), Black Swans and the Domains of Statistics, American Statistician, August 2007, Vol. 6I, №3.

Soufiane Hayou et al. (2019), On the Impact of the Activation Function on Deep Neural Networks Training, Proceedings of the 36 th International Conference on Machine Learning, Long Beach, California, PMLR 97, 2019

TidarukAreerak (2014), Mathematical Model of Stock Prices via a Fractional Brownian Motion Model with Adaptive Parameters

Ton Dieker (2004), Simulation of Fractional Brownian Motion, Thesis, University of Twente, Department of Mathematical Sciences, P.O. BOX 217, 7500 AE Enschede, Netherlands

Wenyu Zhang (2015), Introduction to Ito’s Lemma, Lecture Note, Cornell University, Department of Statistical Sciences, May 6, 2015.

https://www.stoodnt.com/blog/scopes-of-machine-learning-and-artificial-intelligence-in-banking-financial-services-ml-ai-the-future-of-fintechs/

https://medium.com/datadriveninvestor/neural-networks-activation-functions-e371202b56ff

https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/

http://www.datastuff.tech/machine-learning/why-do-neural-networks-need-an-activation-function/

https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0

https://www.youthkiawaaz.com/2019/07/future-of-artificial-intelligence-in-banks/

https://news.efinancialcareers.com/uk-en/328299/ai-in-trading-buy-side

https://ai.stackexchange.com/questions/7609/is-nassim-taleb-right-about-ai-not-being-able-to-accurately-predict-certain-type/7610

https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c

The U.S. Hearing Youtube (Technology Companies & Algorithms): https://www.youtube.com/watch?v=vtw4e68CkwU

https://www.commerce.senate.gov/2019/6/optimizing-for-engagement-understanding-the-use-of-persuasive-technology-on-internet-platforms

https://ai.stackexchange.com/questions/7088/how-to-choose-an-activation-function

https://mlfromscratch.com/activation-functions-explained/#/

https://github.com/nadavo/mood.

Backward propagation and Activation Functions:

https://www.youtube.com/watch?v=q555kfIFUCM

https://www.youtube.com/watch?v=-7scQpJT7uo

https://towardsdatascience.com/analyzing-different-types-of-activation-functions-in-neural-networks-which-one-to-prefer-e11649256209

http://vision.stanford.edu/teaching/cs231n-demos/linear-classify/

http://vision.stanford.edu/teaching/cs231n-demos/knn/

https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.41357&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false