Understanding Deep Neural Networks Training Course

Part 1 – Deep Learning and DNN Concepts

Introduction to AI, Machine Learning & Deep Learning

• History, core concepts, and common applications of artificial intelligence, moving beyond popular misconceptions
• Collective Intelligence: aggregating knowledge shared among numerous virtual agents
• Genetic algorithms: evolving populations of virtual agents through selection processes
• Standard Learning Machines: definitions and scope.
• Task types: supervised learning, unsupervised learning, and reinforcement learning
• Action types: classification, regression, clustering, density estimation, and dimensionality reduction
• Examples of machine learning algorithms: Linear Regression, Naive Bayes, and Random Trees
• Machine Learning vs. Deep Learning: identifying problems where traditional ML remains the state of the art (e.g., Random Forests & XGBoost)

Basic Concepts of Neural Networks (Application: Multi-layer Perceptron)

• Review of essential mathematical foundations.
• Defining neural networks: classic architectures, activation functions, and
• weighting of preceding activations, and network depth
• Defining network learning: cost functions, back-propagation, Stochastic Gradient Descent, and maximum likelihood estimation.
• Modeling neural networks: adapting input and output data handling based on the problem type (regression, classification, etc.). The curse of dimensionality.
• Distinguishing between multi-feature data and signals. Selecting appropriate cost functions based on data characteristics.
• Function approximation by neural networks: theory and practical examples
• Distribution approximation by neural networks: theory and practical examples
• Data Augmentation: techniques for balancing datasets
• Generalizing the results of neural networks.
• Initializing and regularizing neural networks: L1/L2 regularization and Batch Normalization
• Optimization algorithms and convergence strategies

Standard ML / DL Tools

This section provides a comparative overview of key tools, highlighting their advantages, disadvantages, position in the ecosystem, and typical use cases.

• Data management tools: Apache Spark and Apache Hadoop Tools
• Machine Learning libraries: NumPy, SciPy, and Scikit-learn
• High-level deep learning frameworks: PyTorch, Keras, and Lasagne
• Low-level deep learning frameworks: Theano, Torch, Caffe, and TensorFlow

Convolutional Neural Networks (CNN).

• Overview of CNNs: core principles and key applications
• Basic CNN operations: convolutional layers and the use of kernels
• Padding and stride, feature map generation, and pooling layers. Extensions to 1D, 2D, and 3D convolutions.
• Overview of prominent CNN architectures that have set the standard in classification tasks
• Image recognition models: LeNet, VGG Networks, Network in Network, Inception, and ResNet. Analysis of the innovations introduced by each architecture and their broader applications (e.g., 1x1 convolutions or residual connections)
• Implementing attention mechanisms.
• Applying CNNs to common classification tasks (text or image)
• Generative applications of CNNs: super-resolution and pixel-to-pixel segmentation. Overview of
• primary strategies for enhancing feature maps in image generation.

Recurrent Neural Networks (RNN).

• Overview of RNNs: core principles and applications.
• Basic RNN operations: hidden state activations, back-propagation through time, and the unfolded network version.
• Advancements towards Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) networks.
• Overview of various states and architectural evolutions
• Addressing convergence issues and vanishing gradients
• Classic architectures: time-series prediction, classification, and more ...
• RNN Encoder-Decoder architectures. Incorporating attention mechanisms.
• NLP applications: word/character encoding and machine translation.
• Video applications: predicting the next frame in a video sequence.

Generative Models: Variational AutoEncoder (VAE) and Generative Adversarial Networks (GAN).

• Overview of generative models and their relationship with CNNs
• Auto-encoders: dimensionality reduction and constrained generation
• Variational Auto-encoders: generative modeling and distribution approximation. Definition and application of latent space. The reparameterization trick. Applications and observed limitations
• Generative Adversarial Networks: foundational concepts.
• Dual network architecture (Generator and Discriminator) with alternating training phases and available cost functions.
• GAN convergence dynamics and common challenges.
• Enhanced convergence methods: Wasserstein GAN and Began. The Earth Mover's Distance.
• Applications in image and photo generation, text synthesis, and super-resolution.

Deep Reinforcement Learning.

• Overview of reinforcement learning: controlling an agent within a defined environment
• characterized by states and possible actions
• Using neural networks to approximate state-value functions
• Deep Q-Learning: experience replay and its application to controlling video games.
• Policy optimization. On-policy and off-policy methods. Actor-Critic architecture. A3C.
• Applications: controlling individual video games or digital systems.

Part 2 – Theano for Deep Learning

Theano Fundamentals

• Introduction
• Installation and configuration

Theano Functions

• inputs, outputs, updates, and givens

Training and Optimizing Neural Networks with Theano

• Modeling Neural Networks
• Logistic Regression
• Implementing Hidden Layers
• Network Training Procedures
• Computation and Classification
• Optimization Techniques
• Log Loss

Model Testing

Part 3 – DNN using TensorFlow

TensorFlow Fundamentals

• Creating, initializing, saving, and restoring TensorFlow variables
• Feeding, reading, and preloading data in TensorFlow
• Leveraging TensorFlow infrastructure for large-scale model training
• Visualizing and evaluating models using TensorBoard

TensorFlow Mechanics

• Preparing Data
• Downloading datasets
• Defining Inputs and Placeholders
• Constructing the Graph
  • Inference
  • Loss Calculation
  • Training Step
• Training the Model
  • Understanding the Graph
  • Managing the Session
  • Implementing the Training Loop
• Evaluating the Model
  • Building the Evaluation Graph
  • Extracting Evaluation Output

The Perceptron

• Activation functions
• The Perceptron Learning Algorithm
• Binary classification using the Perceptron
• Document classification using the Perceptron
• Limitations of the Perceptron

From Perceptron to Support Vector Machines

• Kernels and the kernel trick
• Maximum margin classification and support vectors

Artificial Neural Networks

• Nonlinear decision boundaries
• Feedforward and feedback artificial neural networks
• Multilayer perceptrons
• Minimizing the cost function
• Forward propagation
• Back-propagation
• Enhancing neural network learning processes

Convolutional Neural Networks

• Learning Objectives
• Model Architecture
• Core Principles
• Code Structure and Organization
• Launching and Training the Model
• Model Evaluation

Basic Introductions to the Following Modules (Brief overview provided based on time availability):

TensorFlow - Advanced Usage

• Threading and Queues
• Distributed TensorFlow
• Writing documentation and sharing models
• Customizing Data Readers
• Manipulating TensorFlow Model Files

TensorFlow Serving

• Introduction
• Basic Serving Tutorial
• Advanced Serving Tutorial
• Serving the Inception Model Tutorial