Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Python Deep Learning Cookbook
Python Deep Learning Cookbook

Python Deep Learning Cookbook: Over 75 practical recipes on neural network modeling, reinforcement learning, and transfer learning using Python

eBook
€8.99 €29.99
Paperback
€36.99
Subscription
Free Trial
Renews at €18.99p/m

What do you get with Print?

Product feature icon Instant access to your digital eBook copy whilst your Print order is Shipped
Product feature icon Paperback book shipped to your preferred address
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Shipping Address

Billing Address

Shipping Methods
Table of content icon View table of contents Preview book icon Preview Book

Python Deep Learning Cookbook

Introduction


The focus of this chapter is to provide solutions to common implementation problems for FNN and other network topologies. The techniques discussed in this chapter also apply to the following chapters.

FNNs are networks where the information only moves in one direction and does not cycle (as we will see in Chapter 4, Recurrent Neural Networks). FNNs are mainly used for supervised learning where the data is not sequential or time-dependent, for example for general classification and regression tasks. We will start by introducing a perceptron and we will show how to implement a perceptron with NumPy. A perceptron demonstrates the mechanics of a single unit. Next, we will increase the complexity by increasing the number of units and introduce single-layer and multi-layer neural networks. The high number of units, in combination with a high number of layers, gives the depth of the architecture and is responsible for the name deep learning. 

Understanding the perceptron


First, we need to understand the basics of neural networks. A neural consists of one or multiple layers of neurons, named after the neurons in human brains. We will demonstrate the mechanics of a single neuron by implementing a perceptron. In a perceptron, a single unit (neuron) performs all the computations. Later, we will scale the number of units to create deep neural networks:

Figure 2.1: Perceptron

A can have multiple inputs. On these inputs, the unit performs some computations and outputs a single value, for example a binary value to classify two classes. The computations performed by the unit are a simple matrix multiplication of the input and the weights. The resulting values are summed up and a bias is added:

These computations can easily be scaled to high dimensional input. An activation function (φ) determines the final output of the in the forward pass:

The weights and bias are initialized. After each epoch (iteration over the training data), the...

Implementing a single-layer neural network


Now we can move on to neural networks. We will start by the simplest form of a neural network: a single-layer neural network. The difference from a perceptron is that the computations are done by multiple units (neurons), hence a network. As you may expect, adding more units will increase the number of problems that can be solved. The units perform their computations separately and are in a layer; we call this layer the hidden layer. Therefore, we call the units in this layer the hidden units. For now, we will only consider a single hidden layer. The output layer performs as a perceptron. This time, as input we have the hidden units in the hidden layer instead of the input variables:

Figure 2.4: Single-layer neural network with two input variables, n hidden units, and a single output unit

In our implementation of the perceptron, we've used a unit step function to determine the class. In the next recipe, we will use a non-linear activation function...

Building a multi-layer neural network


What we've created in the recipe is actually the simplest form of an FNN: a neural network where the information flows only in one direction. For our next recipe, we will extend the number of hidden layers from one to multiple layers. Adding additional layers increases the power of a network to learn complex non-linear patterns. 

Figure 2.7: Two-layer neural network with i input variables, n hidden units, and m hidden units respectively, and a single output unit

As you can see in Figure 2-7, by adding an additional layer the number of connections (weights), also called trainable parameters, increases exponentially. In the next recipe, we will create a network with two hidden layers to predict wine quality. This is a regression task, so we will be using a linear activation for the output layer. For the hidden layers, we use ReLU activation functions. This recipe uses the Keras framework to implement the feed-forward network.

How to do it...

  1. We start by import...

Getting started with activation functions


If we only use linear activation functions, a neural network would represent a large collection of linear combinations. However, the power of neural networks lies in their ability to model complex nonlinear behavior. We briefly introduced the non-linear activation functions sigmoid and ReLU in the previous recipes, and there are many more popular nonlinear functions, such as ELULeaky ReLU, TanH, and Maxout.

There is no rule as to activation works best for the units. Deep learning is a new field and most results are obtained by trial and error instead of mathematical proofs. For the output unit, we use a single output unit and a linear activation for regression tasks. For classification tasks with n classes, we use n output nodes and a softmax activation function. The softmax function forces the network to output probabilities between 0 and 1 for mutually exclusive classes and the probabilities sum up to 1. For binary classification, we can...

Experiment with hidden layers and hidden units


The most commonly used layers in neural networks are fully-connected layers. In fully-connected layers, the units in two successive layers are all  connected. However, the units within a layer don't share any connections. As stated before, the connections between the layers are also called trainable parameters. The weights of these connections are trained by the network. The more connections, the more parameters and the more complex patterns can be modeled. Most state-of-the-art models have 100+ million parameters. However, a deep neural network with many layers and units takes more time to train. Also, with extremely deep models the time to infer predictions takes significantly longer (which can be problematic in a real-time environment). In the following chapters, we will introduce other popular layer types that are specific to their network types. 

Picking the correct number of hidden layers and hidden units can be important. When using too...

Implementing an autoencoder


For autoencoders, we use a network architecture, as shown in the following figure. In the first couple of layers, we decrease the number of hidden units. Halfway, we start increasing the number of hidden units again until the number of hidden units is the same as the number of input variables. The middle hidden layer can be seen as an encoded variant of the inputs, where the output determines the quality of the encoded variant:

Figure 2.13: Autoencoder network with three hidden layers, with m < n

In the next recipe, we will implement an in Keras to decode Street View House Numbers (SVHN) from 32 x 32 images to 32 floating numbers. We can determine the quality of the encoder by decoding back to 32 x 32 and comparing the images.

How to do it...

  1. Import the necessary libraries with the following code:
import numpy as np
from matplotlib import pyplot as plt
import scipy.io

from keras.models import Sequential
from keras.layers.core import Dense
from keras.optimizers...

Tuning the loss function


While training a neural network for a learning problem, the objective of the network is to minimize the loss function. The loss function — also known as error, cost function, or opimization function–compares the prediction with the ground truth during the forward pass. The output of this loss function is used to optimize the weights during the backward pass. Therefore, the loss function is crucial in training the network. By setting the correct loss function, we force the network to optimize towards the desired predictions. For example, for imbalanced datasets we need a different loss function.

In the previous recipes, we've used mean squared error (MSE) and categorical entropy as loss functions. There are also other popular loss functions, and another option is to create a custom loss function. A custom loss function gives the ability to optimize to the desired output. This will be important when we will implement Generative Adversarial Networks (GANs). In the...

Experimenting with different optimizers


The most popular and well optimizer is Stochastic Gradient Descent (SGD). This technique is widely used in other machine learning models as well. SGD is a to find minima or maxima by iteration. There are many popular variants of SGD that try to speed up convergence and less tuning by using an adaptive learning rate. The following table is an overview of the most commonly used optimizers in deep learning:

Optimizer

Hyperparameters

Comments

SGD

Learning rate, decay

+ Learning directly impacts performance (smaller learning rate avoids local minima)

- Requires more manual tuning

- Slow convergence

AdaGrad

Learning rate, epsilon, decay

+ Adaptive learning for all parameters (well suited for sparse data)

- Learning becomes too small and stops learning

AdaDelta

Learning rate, rho, epsilon, decay

+ Faster convergence at start

- Slows near minimum

Adam

Learning rate, beta 1, beta 2, epsilon, decay

+ Adaptive learning rate and momentum for all parameters

RMSprop

Learning rate...

Improving generalization with regularization


Overfitting on the data is one of the biggest of machine learning. There are many machine learning algorithms that are able to train on the training data by remembering all cases. In this scenario, the algorithm might not be able to generalize and make a correct prediction on new data. This is an especially big threat for deep learning, where neural networks have large numbers of trainable parameters. Therefore, it is extremely important to create a representative validation set. 

Note

In deep learning, the general advice when tackling new problems is to overfit as much as you can on the training data first. This ensures that your model is able to train on the training data and is complex enough. Afterwards, you should regularize as much as you can to make sure the model is able to generalize on unseen data (the validation set) as well. 

Most of the techniques used to prevent overfitting can be placed under regularization. Regularization include...

Adding dropout to prevent overfitting


Another popular method for regularization is dropout. A forces a neural network to learn multiple independent representations by randomly removing connections between neurons in the learning phase. For example, when using a dropout of 0.5, the network has to see each example twice before the connection is learned. Therefore, a network with dropout can be seen as an ensemble of networks. 

In the following recipe, we will improve a model that clearly overfits the training data by adding dropouts.

How to do it...

  1. Import the as follows:
import numpy as np 
import pandas as pd
from sklearn.model_selection import train_test_split

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline


import numpy as np...

Adding dropout to prevent overfitting

Another popular method for regularization is dropout. A dropout forces a neural network to learn multiple independent representations by randomly removing connections between neurons in the learning phase. For example, when using a dropout of 0.5, the network has to see each example twice before the connection is learned. Therefore, a network with dropout can be seen as an ensemble of networks. 

In the following recipe, we will improve a model that clearly overfits the training data by adding dropouts.

How to do it...

  1. Import the libraries as follows:
import numpy as np 
import pandas as pd
from sklearn.model_selection import train_test_split

from keras.models import...
Left arrow icon Right arrow icon
Download code icon Download Code

Key benefits

  • - Practical recipes on training different neural network models and tuning them for optimal performance
  • -Use Python frameworks like TensorFlow, Caffe, Keras, Theano for Natural Language Processing, Computer Vision, and more
  • -A hands-on guide covering the common as well as the not so common problems in deep learning using Python

Description

Deep Learning is revolutionizing a wide range of industries. For many applications, deep learning has proven to outperform humans by making faster and more accurate predictions. This book provides a top-down and bottom-up approach to demonstrate deep learning solutions to real-world problems in different areas. These applications include Computer Vision, Natural Language Processing, Time Series, and Robotics. The Python Deep Learning Cookbook presents technical solutions to the issues presented, along with a detailed explanation of the solutions. Furthermore, a discussion on corresponding pros and cons of implementing the proposed solution using one of the popular frameworks like TensorFlow, PyTorch, Keras and CNTK is provided. The book includes recipes that are related to the basic concepts of neural networks. All techniques s, as well as classical networks topologies. The main purpose of this book is to provide Python programmers a detailed list of recipes to apply deep learning to common and not-so-common scenarios.

Who is this book for?

This book is intended for machine learning professionals who are looking to use deep learning algorithms to create real-world applications using Python. Thorough understanding of the machine learning concepts and Python libraries such as NumPy, SciPy and scikit-learn is expected. Additionally, basic knowledge in linear algebra and calculus is desired.

What you will learn

  • • Implement different neural network models in Python
  • • Select the best Python framework for deep learning such as PyTorch, Tensorflow, MXNet and Keras
  • • Apply tips and tricks related to neural networks internals, to boost learning performances
  • • Consolidate machine learning principles and apply them in the deep learning field
  • • Reuse and adapt Python code snippets to everyday problems
  • • Evaluate the cost/benefits and performance implication of each discussed solution
Estimated delivery fee Deliver to Malta

Premium delivery 7 - 10 business days

€32.95
(Includes tracking information)

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Oct 27, 2017
Length: 330 pages
Edition : 1st
Language : English
ISBN-13 : 9781787125193
Vendor :
Google
Category :
Languages :
Concepts :
Tools :

What do you get with Print?

Product feature icon Instant access to your digital eBook copy whilst your Print order is Shipped
Product feature icon Paperback book shipped to your preferred address
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Shipping Address

Billing Address

Shipping Methods
Estimated delivery fee Deliver to Malta

Premium delivery 7 - 10 business days

€32.95
(Includes tracking information)

Product Details

Publication date : Oct 27, 2017
Length: 330 pages
Edition : 1st
Language : English
ISBN-13 : 9781787125193
Vendor :
Google
Category :
Languages :
Concepts :
Tools :

Packt Subscriptions

See our plans and pricing
Modal Close icon
€18.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
€189.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just €5 each
Feature tick icon Exclusive print discounts
€264.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just €5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total 115.97
Python Deep Learning Cookbook
€36.99
Python Machine Learning, Second Edition
€32.99
Python Deep Learning
€45.99
Total 115.97 Stars icon
Banner background image

Table of Contents

14 Chapters
Programming Environments, GPU Computing, Cloud Solutions, and Deep Learning Frameworks Chevron down icon Chevron up icon
Feed-Forward Neural Networks Chevron down icon Chevron up icon
Convolutional Neural Networks Chevron down icon Chevron up icon
Recurrent Neural Networks Chevron down icon Chevron up icon
Reinforcement Learning Chevron down icon Chevron up icon
Generative Adversarial Networks Chevron down icon Chevron up icon
Computer Vision Chevron down icon Chevron up icon
Natural Language Processing Chevron down icon Chevron up icon
Speech Recognition and Video Analysis Chevron down icon Chevron up icon
Time Series and Structured Data Chevron down icon Chevron up icon
Game Playing Agents and Robotics Chevron down icon Chevron up icon
Hyperparameter Selection, Tuning, and Neural Network Learning Chevron down icon Chevron up icon
Network Internals Chevron down icon Chevron up icon
Pretrained Models Chevron down icon Chevron up icon

Customer reviews

Rating distribution
Full star icon Full star icon Full star icon Half star icon Empty star icon 3.7
(3 Ratings)
5 star 66.7%
4 star 0%
3 star 0%
2 star 0%
1 star 33.3%
Gert Dec 30, 2017
Full star icon Full star icon Full star icon Full star icon Full star icon 5
positive points:• The book builds on simple recipes toward more complex recipes.• Great book if you want to learn by example!• A lot of useful code is included in the book that you can reuse for different projects.• What I like is that the author focusses on experimentation and doesn’t assume one method is better over another.negative points:• Sometimes a bit more explanation why some of the choices have been made would be good.• It would be better if the images are in colour (especially some charts).
Amazon Verified review Amazon
newby19 Nov 19, 2017
Full star icon Full star icon Full star icon Full star icon Full star icon 5
I think this book is a great way to get started using deep learning in a hands-on way. Especially for someone who's relatively new to deep learning and wants to experiment and work on different projects.Each recipe in the book is broken down to the different steps to take. Each step is described shortly and to the point. I used the recipes from the Computer Vision chapter right away to create my own image classification project.
Amazon Verified review Amazon
Some Python Guy Dec 19, 2017
Full star icon Empty star icon Empty star icon Empty star icon Empty star icon 1
Found the code files on git but for example the wav/video files from chapter 9 are missing. Likely other source data files are missing. Please provide all relevant input files to run the samples.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is the delivery time and cost of print book? Chevron down icon Chevron up icon

Shipping Details

USA:

'

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.


Unfortunately, due to several restrictions, we are unable to ship to the following countries:

  1. Afghanistan
  2. American Samoa
  3. Belarus
  4. Brunei Darussalam
  5. Central African Republic
  6. The Democratic Republic of Congo
  7. Eritrea
  8. Guinea-bissau
  9. Iran
  10. Lebanon
  11. Libiya Arab Jamahriya
  12. Somalia
  13. Sudan
  14. Russian Federation
  15. Syrian Arab Republic
  16. Ukraine
  17. Venezuela
What is custom duty/charge? Chevron down icon Chevron up icon

Customs duty are charges levied on goods when they cross international borders. It is a tax that is imposed on imported goods. These duties are charged by special authorities and bodies created by local governments and are meant to protect local industries, economies, and businesses.

Do I have to pay customs charges for the print book order? Chevron down icon Chevron up icon

The orders shipped to the countries that are listed under EU27 will not bear custom charges. They are paid by Packt as part of the order.

List of EU27 countries: www.gov.uk/eu-eea:

A custom duty or localized taxes may be applicable on the shipment and would be charged by the recipient country outside of the EU27 which should be paid by the customer and these duties are not included in the shipping charges been charged on the order.

How do I know my custom duty charges? Chevron down icon Chevron up icon

The amount of duty payable varies greatly depending on the imported goods, the country of origin and several other factors like the total invoice amount or dimensions like weight, and other such criteria applicable in your country.

For example:

  • If you live in Mexico, and the declared value of your ordered items is over $ 50, for you to receive a package, you will have to pay additional import tax of 19% which will be $ 9.50 to the courier service.
  • Whereas if you live in Turkey, and the declared value of your ordered items is over € 22, for you to receive a package, you will have to pay additional import tax of 18% which will be € 3.96 to the courier service.
How can I cancel my order? Chevron down icon Chevron up icon

Cancellation Policy for Published Printed Books:

You can cancel any order within 1 hour of placing the order. Simply contact customercare@packt.com with your order details or payment transaction id. If your order has already started the shipment process, we will do our best to stop it. However, if it is already on the way to you then when you receive it, you can contact us at customercare@packt.com using the returns and refund process.

Please understand that Packt Publishing cannot provide refunds or cancel any order except for the cases described in our Return Policy (i.e. Packt Publishing agrees to replace your printed book because it arrives damaged or material defect in book), Packt Publishing will not accept returns.

What is your returns and refunds policy? Chevron down icon Chevron up icon

Return Policy:

We want you to be happy with your purchase from Packtpub.com. We will not hassle you with returning print books to us. If the print book you receive from us is incorrect, damaged, doesn't work or is unacceptably late, please contact Customer Relations Team on customercare@packt.com with the order number and issue details as explained below:

  1. If you ordered (eBook, Video or Print Book) incorrectly or accidentally, please contact Customer Relations Team on customercare@packt.com within one hour of placing the order and we will replace/refund you the item cost.
  2. Sadly, if your eBook or Video file is faulty or a fault occurs during the eBook or Video being made available to you, i.e. during download then you should contact Customer Relations Team within 14 days of purchase on customercare@packt.com who will be able to resolve this issue for you.
  3. You will have a choice of replacement or refund of the problem items.(damaged, defective or incorrect)
  4. Once Customer Care Team confirms that you will be refunded, you should receive the refund within 10 to 12 working days.
  5. If you are only requesting a refund of one book from a multiple order, then we will refund you the appropriate single item.
  6. Where the items were shipped under a free shipping offer, there will be no shipping costs to refund.

On the off chance your printed book arrives damaged, with book material defect, contact our Customer Relation Team on customercare@packt.com within 14 days of receipt of the book with appropriate evidence of damage and we will work with you to secure a replacement copy, if necessary. Please note that each printed book you order from us is individually made by Packt's professional book-printing partner which is on a print-on-demand basis.

What tax is charged? Chevron down icon Chevron up icon

Currently, no tax is charged on the purchase of any print book (subject to change based on the laws and regulations). A localized VAT fee is charged only to our European and UK customers on eBooks, Video and subscriptions that they buy. GST is charged to Indian customers for eBooks and video purchases.

What payment methods can I use? Chevron down icon Chevron up icon

You can pay with the following card types:

  1. Visa Debit
  2. Visa Credit
  3. MasterCard
  4. PayPal
What is the delivery time and cost of print books? Chevron down icon Chevron up icon

Shipping Details

USA:

'

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.


Unfortunately, due to several restrictions, we are unable to ship to the following countries:

  1. Afghanistan
  2. American Samoa
  3. Belarus
  4. Brunei Darussalam
  5. Central African Republic
  6. The Democratic Republic of Congo
  7. Eritrea
  8. Guinea-bissau
  9. Iran
  10. Lebanon
  11. Libiya Arab Jamahriya
  12. Somalia
  13. Sudan
  14. Russian Federation
  15. Syrian Arab Republic
  16. Ukraine
  17. Venezuela