Source: Deep Learning on Medium
The Modules today are the Brain Tomorrow.
What’s the difference on arrival of TensorFlow 2.0?
From TensorFlow 1.x to transition for the TensorFlow 2.0 is going to be a hard one it also the rise of the swift for tensorflow in the same arena its also being supported in the different compilers we can also to deploy for different architecture, but with the right approach in understanding mathematics and the deployed code(back-end scripts) you’ll be able to understand the semantics and with ease have the support for the migration of the code in the tensorflow version.To learn the difference between Keras, tf.keras, and TensorFlow 2.0, Just take a course on the practical machine learning with Tensorflow from nptel or studey from the documentations that is hyperlinked.
What’s the difference in TensorFlow 2.0?
If we look at where we are today the entire world of the Deep learning and the Machine(this was not the same twenty years ago) learning revolves Around the versions for the computing aspects as it algorithm which is the key for it not the code code can be even written by statisticians but the Understanding the code and deploying the actual aspects is the task for all the developers.
- Sessions (@tf.function will do the job).
- eager execution
- Automatic differentiation.
- Model construction.
- layer subclassing.
- Better multi-GPU/distributed training support.
- Support for the various hardware platforms
- Support for the CUDA enabled GPU cards and the deployments of the trained models in the various processes and the entities(this is an enhanced feature for the Hardware deployment).
- Easy description of the architecture by coding.
Included in TensorFlow 2.0 is a complete ecosystem comprised of TensorFlow-Lite (for mobile and embedded devices) and TensorFlow Extended for development production machine learning pipelines (for deploying production models) tf.session and the session.run commands that bulky will not be seen today might be as they will rule the tech today for this too was made for the cloud thus deploying was made a little tough for the execution. I’ve seen a lot of confusion over the rules of tf.Graph and tf.Session in TensorFlow. It’s simple:
- A graph defines the computation. It doesn’t compute anything, it doesn’t hold any values, it just defines the operations that you specified in your code.
- A session allows to execute graphs or part of graphs. It allocates resources (on one or more machines) for that and holds the actual values of intermediate results and variables.
Defining the Graph
We define a graph with a variable and three operations: variable returns the current value of our variable. initialize assigns the initial value of 42 to that variable. assign assigns the new value of 13 to that variable.
On a side note: TensorFlow creates a default graph for you, so we don’t need the first two lines of the code above. The default graph is also what the sessions in the next section use when not manually specifying a graph.just press on the link there or just hold your cursor on the function for few seconds or do ctrl+click for the function.
Running Computations in a Session
To run any of the three defined operations, we need to create a session for that graph. The session will also allocate memory to store the current value of the variable it can be an array or a tensor variable as defined from the tensorflow variable.
As you can see, the value of our variable is only valid within one session. If we try to query the value afterwards in a second session, TensorFlow will raise an error because the variable is not initialized there.
The error https://www.google.com/search?q=FailedPreconditionError%3A%20Attempting%20to%20use%20uninitialized%20value%20foo%0A%09%20%5B%5B%7B%7Bnode%20_retval_foo_0_0%7D%7D%5D%5D+site:stackoverflow.com was not available even after the search.
Of course, we can use the graph in more than one session, we just have to initialize the variables again. The values in the new session will be completely independent from the first one:
Hopefully this short walk through helped you to better understand tf.Session. Feel free to ask questions in the comments.
The intertwined relationship between Keras and TensorFlow
- Keras was originally created and developed by Google AI Developer/Researcher, Francois Chollet.
- Francois committed and released the first version of Keras to his GitHub on March 27th, 2015 from then on there is no set back you can also read its documentation in the keras.io.
- Initially, Francois developed Keras to facilitate his own research and experiments However, with the explosion of deep learning popularity, many developers, programmers, and machine learning practitioners flocked to Keras due to its easy-to-use API as it is developer friendly.
- Back then, there weren’t too many deep learning libraries available — the popular ones included Torch(Github of Theano), Theano(developed by montreal institute of learning algorithm(MILA)), and Caffe(Caffe is developed by Yangqing Jia)
- The problem with these libraries was that it was like trying to write assembly/C++ to perform your experiments — tedious, time-consuming, and inefficient.
- Keras, on the other hand, was extremely easy to use, making it possible for researchers and developers to iterate on their experiments faster.
- In order to train your own custom neural networks, Keras required a backend.
- A backend is a computational engine — it builds the network graph/topology, runs the optimizers, and performs the actual number crunching.
- To understand the concept of a backend, consider building a UI like tkinter. A code is always used to interact with the database will not change (provided you’re using some sort of MVC paradigm that abstracts the database layer, of course). Essentially, code has to acess what database is being used, as long as developer knows the compiler and the purpopse.The same is true with Keras. You can think of the backend as your database and Keras as your programming language used to access the database. You can swap in whatever backend you like, and as long as it abides by certain rules, your code doesn’t have to change.Therefore, you can think of Keras as a set of abstractions that makes it easier to perform deep learning
- Originally, Keras’ default backend was Theano and was the default until v1.10.0
- At the same time, Google had released TensorFlow, a symbolic math library used for machine learning and training neural networks.
- Once TensorFlow became the default backend for Keras, by definition, both TensorFlow and Keras usage grew together — you could not have Keras without TensorFlow, and if you installed Keras on your system, you were also installing TensorFlow.
- The tf . keras submodule was introduced in TensorFlow v1, the first step in integrating Keras directly within the TensorFlow package itself.
- The tf . keras package is/was separate from the keras package you would install via pip (i.e., pip install keras ).
- The original keras package was not subsumed into tensorflow to ensure compatibility and so that they could both organically develop.
- However, that’s now changing — when Google announced TensorFlow 2.0 in June 2019, they declared that Keras is now the official high-level API of TensorFlow for quick and easy model design and training.
- With the release of Keras 2.3.0, Francois has stated that:
- This is the first release of Keras that brings the keras package in sync with tf . keras
- It is the final release of Keras that will support multiple backends (i.e., Theano, CNTK, etc.).
- And most importantly, going forward all deep learning practitioners should switch their code to TensorFlow 2.0 and the tf . keras package.
- The original keras package will still receive bug fixes, but moving forward, you should be using tf . keras .
Start using tf.keras in all future project
Keras v2.3.0 was officially released — in the release Francois Chollet (the creator and chief maintainer of Keras), stated that Keras v2.3.0 is the first release of Keras that brings keras in sync with It will be the the last major release to support backends other than TensorFlow (i.e., Theano, CNTK, etc.)
And most importantly, deep learning practitioners should start moving to TensorFlow 2.0 and the tf . keras package(similar to keras but the actual tensorflow performance instance is added) For the majority of your projects, that’s as simple as changing your import lines from
To help you in (automatically) updating your code from keras to , Google has released a script named tf_upgrade_v2 script, which, as the name suggests, analyzes your code and reports which lines need to be updated — the script can even perform the upgrade process for you,there were also some api changes mentioned there.
API changes are
- Add size(x) to backend API.
- add_metric method added to Layer / Model (used in a similar way as add_loss, but for metrics), as well as the metrics property.
- Variables set as attributes of a Layer are now tracked in layer.weights (including layer.trainable_weights or layer.non_trainable_weights as appropriate).
- Layers set as attributes of a Layer are now tracked (so the weights/metrics/losses/etc of a sublayer are tracked by parent layers). This behavior already existed for Model specifically and is now extended to all Layer subclasses.
- Introduce class-based losses (inheriting from Loss base class). This enables losses to be parameterized via constructor arguments. Loss classes added:
- Introduce class-based metrics (inheriting from Metric base class). This enables metrics to be stateful (e.g. required for supported AUC) and to be parameterized via constructor arguments. Metric classes added:
- Add reset_metrics argument to train_on_batch and test_on_batch. Set this to True to maintain metric state across different batches when writing lower-level training/evaluation loops. If False, the metric value reported as output of the method call will be the value for the current batch only.
- Add model.reset_metrics() method to Model. Use this at the start of an epoch to clear metric state when writing lower-level training/evaluation loops.
- Rename lr to learning_rate for all optimizers.
- Deprecate argument decay for all optimizers. For learning rate decay, use LearningRateSchedule objects in tf.keras.
Click+ctrl on the highlighted text will give the code above(The code describing the version control)
Keras a configurable in the framework and the supported packages.
As I mentioned earlier in this post, Keras relies on the concept of a computational backend The computational backend performs all the “heavy computations” in terms of constructing a graph of the model, numeric computation, etc. Keras then sits on top of this computational engine as an abstraction, making it easier for deep learning developers to implement their models.Originally, Keras supported Theano as its preferred computational backend — it then later supported other backends, including CNTK and mxnet this is now develpoed by other companies like microsoft ,Google ,NVIDIA and many more companies.However, the most popular backend, is TensorFlow which eventually became the default computation backend for Keras and also tf.keras. TensorFlow v1 was the first release of TensorFlow to include a branch of keras inside Now that TensorFlow 2.0 is released both keras and are in sync with keras however, developers should start moving forward as the keras package will only support bug fixes.
To quote Francois Chollet, the creator and maintainer of Keras:
“This is also the last major release of multi-backend Keras. Going forward, we recommend that users consider switching their Keras code to tf.keras in TensorFlow 2.0.It implements the same Keras 2.3.0 API (so switching should be as easy as changing the Keras import statements), but it has many advantages for TensorFlow users, such as support for eager execution, distribution, TPU training, and generally far better integration between low-level TensorFlow and high-level concepts like Layer and Model.”
Sessions and Eager Execution in TensorFlow 2.0
TensorFlow 1.10+ users that utilize the Keras API within will be familiar with creating a Session to train their model,Creating the Session object and requiring the entire model graph to be built ahead of time was a bit of a pain, so TensorFlow 2.0 introduced the concept of Eager Execution, thereby simplifying the code to the benefit of Eager Execution is that the entire model graph does not have to be built.Instead, operations are evaluated immediately, making it easier to get started building your models (as well as debugging them) infact a graph gets built on the similar fashion in the backend we dont need to code it.in order to get develop you have to ask a question on :
“how to use it with TensorFlow 2.0 and to understand the difference between Eager Execution and Sessions and the impact it has on the speed of training a model and also on the deployment of these models”
Automatic differentiation and GradientTape with TensorFlow 2.0
If we are researchers who are in need to implement custom layers or loss functions keras would help in deploying it.TensorFlow 1.x’s custom implementations were bulky and needed more time for the implementation.With the release of TensorFlow 2.0 that is starting to change — it’s now far easier to implement your own custom losses.One way it’s becoming easier is through automatic differentiation and the GradientTape implementation.To utilize GradientTape all we need to do is implement our model architecture:
Define our loss function and optimizer:
Create the function responsible for performing a single batch update:
And then train the model.
this is just a little steps as an approach for learning and deploying the models and also to cover the different aspects we just need the code for the tensors and train the function to extract the features of any sequential data.The Gradient Tape magic handles differentiation for us behind the scenes, making it easier to work with custom losses and layers and speaking of custom layer and model implementations,based on the calculation made for the users definition and also to deploy them based on papers referd and also from the sources used for building such systems or the architecture.
Model and layer subclassing in TensorFlow 2.0
Both the sequential and functional paradigms have been inside Keras for quite a while, but the Subclassing feature is still unknown to many deep learning practitioners.The call method then performs the forward-pass, enabling us to customize the forward pass as we see fit.
The benefit of using model subclassing is that our model can be:
- fully-customization.(can be a JSON or any other formats to store but it can be configured and can be made as per the users desire)
- Enables us to implement and utilize your own custom loss implementations and train for the feature extraction and also to give the extensive formats for the version control,reliability and reusablity of code.
since our architecture inherits the Model class, we can still call methods like , , and , thereby maintaining the easy-to-use (and familiar) Keras API.If you’re interested in learning more about GoogleNet,mobile net(linked a paper for understanding the architecture and also to understand its application),ConvNet,RNN, baby RNN,LeNet etc.Get to know the input data set preprocess it and deploy it in the neural network architecture as mentioned above.
TensorFlow has tools for optimizing your models for faster inference: https://www.tensorflow.org/model_optimization
This includes post-training weight quantization, and gradual weight pruning during training for your Keras models even trainable machine gives out the models in the tflite for the coral and also for the android thus they can have a migratable models for the access.
TensorFlow 2.0 introduces better multi-GPU and distributed training support
TensorFlow 2.0 and provide better multi-GPU and distributed training through their MirroredStrategy.
To quote the TensorFlow 2.0 documentation, “The MirroredStrategy supports synchronous distributed training on multiple GPUs on one machine”.
If you want to use multiple machines (each having potentially multiple GPUs), you should take a look at the MultiWorkerMirroredStrategy.Or, if you are using Google’s cloud for training, check out the TPUStrategy.For now though, let’s assume you are on a single machine that has multiple GPUs and you want to ensure all of your GPUs are used for training.You can accomplish this by first creating your MirroredStrategy :
You then need to declare your model architecture and compile it within the scope of the strategy :And from there you can call . fit to train the model:Provided your machine has multiple GPUs, TensorFlow will take care of the multi-GPU training for you.
TensorFlow 2.0 is an ecosystem, including TF 2.0, TF Lite, TFX,tfrl,TensorflowJS,tensorflow encrypted and other functional aspects like quantization, and deployment of the production level code and the trained models.
TensorFlow 2.0 is more than a computational engine and a deep learning library for training neural networks — it’s so much more.With TensorFlow Lite (TF Lite) we can train, optimize, and quantize models that are designed to run on resource-constrained devices such as smartphones and other embedded devices (i.e., Raspberry Pi, Google Coral, etc.) if you need to deploy our model to production, you can use TensorFlow Extended (TFX), an end-to-end platform for model deployment.Once our research and experiments are complete, you can leverage TFX to prepare the model for production and scale your model using Google’s ecosystem or any platofrom on which we have deployed the code.With TensorFlow 2.0 we are truly starting to see a better, more efficient bridge between research, experimentation and deployment to production.
All code examples from this post came from TensorFlow 2.0’s official examples they to have deployed the code for diffrent data set and they have the production level code being deployed and thus we have to work on the other aspects of the code and the math by taking this open source platform and making a proper use of and also develop and depoloy such aspects for the problem statement refer to the complete code examples provided by Francois Chollet for more details and also a code that is availble for the resources also see the Google IO for more information as this was a plan for the developers before in google that was being deployed.
Speed and Optimization of TensorFlow 2.0 will be the aspects coverd for the operation of the different levels of the abstraction for algorithm and training and the test data sets, but you’ll also receive new feature updates — the latest release of the keras package version 2.3.0 and the last release to support multiple back ends and feature updates. and upgrades will be useful in online training of the models to be made more easier for the deployment.
TensorFlow 2.0 is more than a GPU-accelerated deep learning library.Not only you will have the ability to train your own models using TensorFlow 2.0 and , but you can now:
- Take those models and prepare them for mobile/embedded deployment using TensorFlow Lite (TF Lite) even teachable machiene will give us the essence to run the models on the edge devices.
- Deploy the models to production using TensorFlow Extended (TF Extended).
We all need to build a computationally stable and more performance oriented for the spatial domain and the architecture has to be deployed for the multiple use of it and also have a great version control for the code and help for the migration.