TensorFlow is one of the popular libraries for developing machine learning and deep neural network models in various domains which is used within Google for research and product development. TensorFlow.js, web version of Tensorflow, is an open-source library that we can use to define, train, and run machine learning models entirely in the browser, using Javascript and a high-level layers API. This tutorial is an introduction to TensorFlow.js and its Core Concepts.
Contents
What is TensorFlow.js?
In-Browser Machine Learning
ML running in the browser means that our program runs on a webpage without need to install any libraries or drivers. TensorFlow.js automatically supports WebGL, and with GPU acceleration, our code can work better behind the scene.
User may also open the webpage from a mobile device, it gives us advantages:
– ML model can take advantage of mobile sensor data
– low-latency data inference and privacy preserving because all data stays on the client
General uses
– Import an existing, pre-trained model for inference
Using model converters, we can convert an existing TensorFlow or Keras model (trained offline) into TensorFlow.js format, then load it into the browser for inference.
– Re-train an imported model
We can augment an existing model trained offline using sensor data connected to the browser, or other client-side data. This way helps to train an accurate model quickly, when we have only a small amount of data.
– Author ML models directly in browser
We can also use TensorFlow.js to define, build, train, and run models entirely from scratch in the browser using the low-level JavaScript linear algebra library or the high-level layers API.
TensorFlow.js API
Tensorflow.js has four layers:
– WebGL API for GPU-supported numerical operations
– Web browser for user interactions
– Low-level Core API (former deeplearn.js library) for hardware-accelerated linear algebra operations and automatic differentiation (Eager)
– Higher-level Layers API for building machine-learning models on top of Core which is modelled after Keras and implements similar functionality.
Tensorflow.js API allows to import Python trained models with Keras or TensorFlow SavedModel and use it for inference or transfer learning in the browser.
TensorFlow.js Core Concepts
Tensor
Tensor is the basic unit and a fundamental data structure in TensorFlow.
A tensor is an n-dimensional collection of numerical values, identified by rank, shape, and type.
– Rank is the number of dimensions of a tensor.
– Shape is the list denoting the size in each dimension.
For example:
+ a scalar is a tensor of rank 0 and thus has a shape of [1]
+ a vector or a one-dimensional array is a tensor of rank 1 and has a shape of [columns] or [rows] or [numberItems]
+ a matrix or a two-dimensional array is a tensor of rank 2 and has a shape of [rows, columns]
+ a three-dimensional array is a tensor of rank 3
+ an n-dimensional array is a tensor of rank n
– Type: at the time of writing this tutorial, TensorFlow.js has 3 data types: float32, int32, bool.
Create Tensor
Common Case
– tf.tensor (values, shape?, dtype?)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
const shape = [2, 3]; const a = tf.tensor([1, 2, 3, 4, 5, 6], shape); a.print(); /* Tensor [[1, 2, 3], [4, 5, 6]] */ const b = tf.tensor([[1, 2], [3, 4], [5, 6]]); b.print(); /* Tensor [[1, 2], [3, 4], [5, 6]] */ |
– tf.scalar (value, dtype?)
|
1 2 3 4 5 6 7 8 9 10 11 |
tf.scalar(3.14, 'int32').print(); /* Tensor 3 */ tf.scalar(3.14, 'float32').print(); /* Tensor 3.140000104904175 */ |
– tf.tensor1d (values, dtype?)
|
1 2 3 4 5 |
tf.tensor1d([1, 2, 3]).print(); /* Tensor [1, 2, 3] */ |
– tf.tensor2d (values, shape?, dtype?)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
tf.tensor2d([[1, 2, 3], [4, 5, 6]]).print(); /* Tensor [[1, 2, 3], [4, 5, 6]] */ tf.tensor2d([1, 2, 3, 4, 5, 6], [3, 2]).print(); /* Tensor [[1, 2], [3, 4], [5, 6]] */ |
– tf.tensor3d (values, shape?, dtype?)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
tf.tensor3d([[[1], [2], [3]], [[4], [5], [6]]]).print(); /* Tensor [[[1], [2], [3]], [[4], [5], [6]]] */ tf.tensor3d([1, 2, 3, 4, 5, 6], [3, 2, 1]).print(); /* Tensor [[[1], [2]], [[3], [4]], [[5], [6]]] */ |
Elements with the same values
– tf.zeros (shape, dtype?): all elements set to 0.
– tf.zerosLike (x): all elements set to 0 with the same shape as the given tensor.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
tf.zeros([3, 5]).print(); /* Tensor [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]] */ const x = tf.tensor([2, 3, 4, 5]); tf.zerosLike(x).print(); /* Tensor [0, 0, 0, 0] */ |
– tf.ones (shape, dtype?): all elements set to 1.
– tf.onesLike (x): all elements set to 1 with the same shape as the given tensor.
– tf.fill (shape, value, dtype?): all elements set to a scalar value.
|
1 2 3 4 5 6 |
tf.fill([2, 3], 8).print(); /* Tensor [[8, 8, 8], [8, 8, 8]] */ |
Elements from sequences
– tf.linspace (start, stop, num): returns Tensor, its elements are evenly spaced sequence of numbers.
|
1 2 3 4 5 6 7 8 9 10 11 |
tf.linspace(1, 9, 9).print(); /* Tensor [1, 2, 3, 4, 5, 6, 7, 8, 9] */ tf.linspace(1, 9, 5).print(); /* Tensor [1, 3, 5, 7, 9] */ |
– tf.range (start, stop, step?, dtype?): returns tf.Tensor1D filled with the numbers in the range(start,stop) provided.
|
1 2 3 4 5 6 7 8 9 10 11 |
tf.range(0, 9).print(); /* Tensor [0, 1, 2, 3, 4, 5, 6, 7, 8] */ tf.range(0, 9, 3).print(); /* Tensor [0, 3, 6] */ |
Elements with random distribution
– tf.randomNormal (shape, mean?, stdDev?, dtype?, seed?): returns Tensor with values sampled from a normal distribution.
+ shape (number[]): output tensor shape
+ mean (number)
+ stdDev (number): standard deviation
+ dtype (‘float32’|’int32’)
+ seed (number): seed for the random number generator
|
1 2 3 4 5 6 |
tf.randomNormal([2, 2], 0, 1).print(); /* Tensor [[-1.1488395, 0.0566379], [1.6311718 , 0.6639878]] */ |
– tf.truncatedNormal (shape, mean?, stdDev?, dtype?, seed?): returns Tensor with values sampled from a truncated normal distribution.
– tf.randomUniform (shape, minval?, maxval?, dtype?): returns Tensor with values sampled from a uniform distribution.
|
1 2 3 4 5 6 |
tf.randomUniform([2, 2], 1, 2).print(); /* Tensor [[1.3520408, 1.7230165], [1.6436907, 1.9786676]] */ |
Variable
While using TensorFlow to build and train models, we need to hold values in a memory location that can be updated/modified during the execution of the program. It is identified by variable in TensorFlow.
We can assign a new tensor to an existing variable with assign() method.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
const tensor = tf.zeros([3]); const tvar = tf.variable(tensor); // initialize tvar tvar.print(); /* Tensor [0, 0, 0] */ // tvar.assign(tf.tensor1d([1, 2, 3, 4, 5])); // => Error: shape of the new value [5] and previous value [3] must match tvar.assign(tf.tensor1d([1, 2, 3])); // update values of tvar tvar.print(); /* Tensor [1, 2, 3] */ |
Operations
TensorFlow.js provides operations that can be performed on tensors.
These operations do not change the tensors’ values but return new tensors.
Arithmetic operations
– tf.add (a, b), tf.sub (a, b), tf.mul (a, b), tf.div (a, b)
– tf.mod (a, b)
– tf.pow (base, exp)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
const x1 = tf.tensor2d([[1, 1], [1, 1]]); const y11 = tf.tensor2d([[2, 2], [2, 2]]); const y12 = tf.scalar(5); tf.add(x1, y11).print(); x1.add(y11).print(); /* Tensor [[3, 3], [3, 3]] */ tf.add(x1, y12).print(); x1.add(y12).print(); /* Tensor [[6, 6], [6, 6]] */ |
– tf.maximum (a, b), tf.minimum (a, b)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
const x2 = tf.tensor1d([2, 4, 3]); const y21 = tf.tensor1d([1, 2, 9]); const y22 = tf.scalar(3); tf.maximum(x2, y21).print(); x2.maximum(y21).print(); /* Tensor [2, 4, 9] */ tf.maximum(x2, y22).print(); x2.maximum(y22).print(); /* Tensor [3, 4, 3] */ |
– tf.squaredDifference (a, b): returns (a – b) * (a – b) element-wise.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
const x3 = tf.tensor1d([2, 4, 3]); const y31 = tf.tensor1d([1, 2, 9]); const y32 = tf.scalar(2); tf.squaredDifference(x3, y31).print(); x3.squaredDifference(y31).print(); /* Tensor [1, 4, 36] */ tf.squaredDifference(x3, y32).print(); x3.squaredDifference(y32).print(); /* Tensor [0, 4, 1] */ |
Basic math operations
– tf.reciprocal (x)
– tf.square (x)
– tf.sqrt (x)
– tf.rsqrt (x): reciprocal of square root
– tf.exp (x)
– tf.log (x)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
const x4 = tf.tensor2d([[-1, 2], [-3, 4]]); tf.square(x4).print(); x4.square().print(); /* Tensor [[1, 4 ], [9, 16]] */ tf.sqrt(x4).print(); x4.sqrt().print(); /* Tensor [[NaN, 1.4142135], [NaN, 2 ]] */ |
– tf.abs (x)
– tf.sign (x)
– tf.neg (x): (-1 * x)
– tf.round (x)
– tf.floor (x)
– tf.ceil (x)
|
1 2 3 4 5 6 7 8 |
const x5 = tf.tensor1d([-1, 2, -3]); tf.abs(x5) x5.abs().print(); /* Tensor [1, 2, 3] */ |
– tf.sigmoid, tf.logSigmoid
– tf.elu (exponential linear), tf.selu (x) (scaled exponential linear)
– tf.relu (x), tf.leakyRelu (x, alpha?), tf.prelu (x, alpha) (leaky rectified linear)
– tf.erf (Gauss error function)
|
1 2 3 4 5 6 7 8 |
const x6 = tf.tensor1d([-1, 2, -3, 4]); tf.relu(x6).print(); x6.relu().print(); /* Tensor [0, 2, 0, 4] */ |
– tf.cos, tf.acos, tf.cosh, tf.acosh
– tf.sin, tf.asin, tf.sinh, tf.asinh
– tf.tan, tf.atan, tf.tanh, tf.atanh
Matrix operations
tf.matMul
tf.matMul (a, b, transposeA?, transposeB?): dot product of two matrices (A * B)
If transpose param is true, the matrx is transposed before multiplication.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
const x7 = tf.tensor2d([1, 2], [1, 2]); /* Tensor [[1, 2],] */ const y7 = tf.tensor2d([1, 2, 3, 4], [2, 2]); /* Tensor [[1, 2], [3, 4]] */ tf.matMul(x7, y7).print(); x7.matMul(y7).print(); /* Tensor [[7, 10],] */ |
tf.outerProduct
tf.outerProduct (v1, v2)
|
1 2 3 4 5 6 7 8 9 10 |
const x8 = tf.tensor1d([1, 2, 3]); const y8 = tf.tensor1d([1, 2, 3]); tf.outerProduct(x8, y8).print(); /* Tensor [[1, 2, 3], [2, 4, 6], [3, 6, 9]] */ |
tf.transpose
tf.transpose (x, perm?): permutes the dimensions according to perm.
The returned Tensor’s dimension i will correspond to the input dimension perm[i].
If perm is not given, it is set to [n-1...0], where n is the rank of the input Tensor.
For example: Input Tensor has shape=[1,3,4].
+ not giving perm: perm=[2,1,0] => returned Tensor has shape=[4,3,1]
+ perm=[0,2,1] => returned Tensor has shape=[1,4,3]
+ perm=[1,2,0] => returned Tensor has shape=[3,4,1]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
const x9 = tf.tensor3d([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], [1, 3, 4]); x9.print(true); /* Tensor rank: 3 shape: [1,3,4] values: [[[1, 2 , 3 , 4 ], [5, 6 , 7 , 8 ], [9, 10, 11, 12]]] */ x9.transpose().print(true); // or tf.transpose(x9) /* Tensor rank: 3 shape: [4,3,1] values: [[[1 ], [5 ], [9 ]], [[2 ], [6 ], [10]], [[3 ], [7 ], [11]], [[4 ], [8 ], [12]]] */ x9.transpose([0, 2, 1]).print(true); /* Tensor rank: 3 shape: [1,4,3] values: [[[1, 5, 9 ], [2, 6, 10], [3, 7, 11], [4, 8, 12]]] */ x9.transpose([1, 2, 0]).print(true); /* Tensor rank: 3 shape: [3,4,1] values: [[[1 ], [2 ], [3 ], [4 ]], [[5 ], [6 ], [7 ], [8 ]], [[9 ], [10], [11], [12]]] */ |
tf.norm
tf.norm (x, ord?, axis?, keepDims?): computes several different vector norms (1-norm, Euclidean or 2-norm, inf-norm, p-norm with p > 0) and matrix norms (Frobenius, 1-norm, inf-norm).
| ord | norm (matrix) | norm (vector) |
|---|---|---|
| ‘euclidean’ | Frobenius norm | 2-norm |
| ‘fro’ | Frobenius norm | |
| Infinity | max(sum(abs(x), axis=1)) | max(abs(x)) |
| -Infinity | min(sum(abs(x), axis=1)) | min(abs(x)) |
| 1 | max(sum(abs(x), axis=0)) | sum(abs(x)) |
| 2 | sum(abs(x)^2)^1/2 |
axis (number|number[]):
+ if axis is null (default): input is considered a vector, norm is computed over the entire set of values in the Tensor.
+ if axis is an integer: input is considered a batch of vectors, and axis determines the axis in x over which to compute vector norms.
+ if axis is a 2-tuple of integer: input is considered a batch of matrices, and axis determines the axes in NDArray over which to compute a matrix norm.
keepDims (boolean): if true, norm has the same dimensionality as the input.
– Example 1:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
// (-3)^2 + 4^2 = 5^2 const x = tf.tensor1d([-3, 4]); x.norm().print(); /* Tensor 5 */ x.norm(2).print(); /* Tensor 5 */ x.norm(1).print(); /* Tensor 7 */ x.norm('euclidean').print(); /* Tensor 5 */ x.norm(Infinity).print(); /* Tensor 4 */ x.norm(-Infinity).print(); /* Tensor 3 */ |
– Example 2:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
// (-3)^2 + 4^2 = 5^2 // (-5)^2 + 12^2 = 13^2 const y = tf.tensor2d([[-3, 4], [-5, 12]]); y.norm().print(); /* Tensor 13.928388595581055 */ y.norm(2).print(); /* Tensor 13.928388595581055 */ y.norm(1).print(); /* Tensor 24 */ y.norm('euclidean').print(); /* Tensor 13.928388595581055 */ y.norm(Infinity).print(); /* Tensor 12 */ y.norm(-Infinity).print(); /* Tensor 3 */ |
– Example 3:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
// (-3)^2 + 4^2 = 5^2 // (-5)^2 + 12^2 = 13^2 const y = tf.tensor2d([[-3, 4], [-5, 12]]); y.norm(2, 0).print(); /* Tensor [5.8309517, 12.6491117] */ y.norm(2, 1).print(); /* Tensor [5, 13] */ y.norm(1, 0).print(); /* Tensor [8, 16] */ y.norm(1, 1).print(); /* Tensor [7, 17] */ y.norm(Infinity, 0).print(); /* Tensor [5, 12] */ y.norm(Infinity, 1).print(); /* Tensor [4, 12] */ y.norm(Infinity, [0, 1]).print(); /* Tensor 17 */ y.norm(-Infinity, 0).print(); /* Tensor [3, 4] */ y.norm(-Infinity, 1).print(); /* Tensor [3, 5] */ y.norm(-Infinity, [0, 1]).print(); /* Tensor 7 */ |
Logical operations
– tf.equal (a, b)
– tf.greater (a, b)
– tf.greaterEqual (a, b)
– tf.less (a, b)
– tf.lessEqual (a, b)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
const a1 = tf.tensor1d([1, 3, 5]); const b11 = tf.tensor1d([1, 2, 3]); const b12 = tf.scalar(4); tf.greater(a1, b11).print(); a1.greater(b11).print(); /* Tensor [0, 1, 1] */ tf.greater(a1, b12).print(); a1.greater(b12).print(); /* Tensor [0, 0, 1] */ |
– tf.logicalAnd (a, b)
– tf.logicalNot (a, b)
– tf.logicalOr (a, b)
– tf.logicalXor (a, b)
|
1 2 3 4 5 6 7 8 9 |
const a2 = tf.tensor1d([1, 1, 0, 0], 'bool'); const b2 = tf.tensor1d([1, 0, 1, 0], 'bool'); tf.logicalAnd(a2, b2).print(); a2.logicalAnd(b2).print(); /* Tensor [1, 0, 0, 0] */ |
– tf.where (condition, a, b): returns Tensor which each element is selected from a if condition is true, b if condition is false.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
const a3 = tf.tensor1d([1, 3, 5, 7]); const b3 = tf.tensor1d([2, 3, 4, 9]); tf.where(tf.greater(a3, b3), a3, b3).print(); /* Tensor [2, 3, 5, 9] */ const a4 = tf.tensor1d([1, 3, 5, 7]); const b4 = tf.scalar(4); const alt = tf.fill([4], 9999); /* Tensor [9999, 9999, 9999, 9999] */ tf.where(tf.less(a4, b4), a4, alt).print(); /* Tensor [1, 3, 9999, 9999] */ |
Models and Layers
Model
A model is just like a function in programming concept: We give it some input, it will produce some desired output.
There are two ways to create a model in TensorFlow.js.
Use operations directly
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
// Define function function predict(input) { // tf.tidy will clean up all the GPU memory used by tensors inside return tf.tidy(() => { const x = tf.scalar(input); // y = a * x ^ 2 + b * x + c const ax2 = a.mul(x.square()); const bx = b.mul(x); const y = ax2.add(bx).add(c); return y; }); } // y = x^2 + 2x + 1 with x = 2 const a = tf.scalar(1); const b = tf.scalar(2); const c = tf.scalar(1); const result = predict(2); result.print(); /* Tensor 9 */ |
Use high-level API
We can think of model as a primary abstraction that consists of layers.
It can be trained, evaluated, and used for prediction.
There are two function for creating models:
– tf.sequential(): returns a tf.Sequential (that extends tf.Model) object, use when outputs of one layer are the inputs to the next layer (supports only a linear stack of layers).
– tf.model(): creates a tf.Model that supports an arbitrary graph (without cycles) of layers.
In TensorFlow.js, tf.Model is the basic unit of training, inference and evaluation.
It has some useful methods:
– compile(): configures and prepares the model for training and evaluation with optimizer, loss, and/or metrics.
– evaluate(): returns the loss value & metrics (that are specified during compile()) values for the model in test mode.
– predict(), predictOnBatch(): generates output predictions for the input/a single batch of samples.
– fit(): trains the model for a fixed number of epochs (iterations on a dataset).
– getLayer(): retrieves a layer by its name or index.
For example, we construct a convolutional image classifier model using tf.sequential:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
const model = tf.sequential(); model.add(tf.layers.conv2d({ inputShape: [28, 28, 1], kernelSize: 5, filters: 8, strides: 1, activation: 'relu', kernelInitializer: 'varianceScaling' })); model.add(tf.layers.maxPooling2d({ poolSize: [2, 2], strides: [2, 2] })); model.add(tf.layers.conv2d({ kernelSize: 5, filters: 16, strides: 1, activation: 'relu', kernelInitializer: 'varianceScaling' })); model.add(tf.layers.maxPooling2d({ poolSize: [2, 2], strides: [2, 2] })); model.add(tf.layers.flatten()); model.add(tf.layers.dense( { units: 10, kernelInitializer: 'varianceScaling', activation: 'softmax' })); const LEARNING_RATE = 0.15; const optimizer = tf.train.sgd(LEARNING_RATE); model.compile({ optimizer: optimizer, loss: 'categoricalCrossentropy', metrics: ['accuracy'], }); |
Layer
We can see that our model above includes a sequence of 6 layers:
– two-dimensional convolutional layer: slide a filter window over an image to learn transformations
– max pooling layer: downsample the result
– second convolutional layer, followed by another pooling layer: repeat layer structure
– flatten layer: output of the previous layer becomes a vector
– dense layer (aka fully connected layer): perform the final classification
So, what is layer?
They are the primary building block for constructing a model.
Each layer gets input, performs some computation, generates output. It automatically creates and initializes the various internal variables/weights for function.
There are many different types of layers available in TensorFlow.js:
Basic
tf.layers.activation
tf.layers.dense
tf.layers.dropout
tf.layers.embedding
tf.layers.flatten
tf.layers.repeatVector
tf.layers.reshape
Convolutional
tf.layers.conv1d
tf.layers.conv2d
tf.layers.conv2dTranspose
tf.layers.depthwiseConv2d
tf.layers.separableConv2d
Merge
tf.layers.add
tf.layers.average
tf.layers.concatenate
tf.layers.maximum
tf.layers.minimum
tf.layers.multiply
Normalization
tf.layers.batchNormalization
Pooling
tf.layers.averagePooling1d
tf.layers.averagePooling2d
tf.layers.globalAveragePooling1d
tf.layers.globalAveragePooling2d
tf.layers.globalMaxPooling1d
tf.layers.globalMaxPooling2d
tf.layers.maxPooling1d
tf.layers.maxPooling2d
Recurrent
tf.layers.gru
tf.layers.gruCell
tf.layers.lstm
tf.layers.lstmCell
tf.layers.rnn
tf.layers.simpleRNN
tf.layers.simpleRNNCell
tf.layers.stackedRNNCells
Advanced Activation
tf.layers.elu
tf.layers.leakyReLU
tf.layers.softmax
tf.layers.thresholdedReLU
For more details, please visit: TensorFlow.js Layers API.
Training
TensorFlow.js also provides API to perform training and compute gradients eagerly.
We provide a function that is a combination of operations and the API automatically differentiate that function’s output with respect to its inputs.
Gradients
tf.grad
tf.grads
tf.customGrad
tf.valueAndGrad
tf.valueAndGrads
tf.variableGrads
For example, gradients of f(a,b):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
// f(a,b) = a*b const f = (a, b) => a.mul(b); // df/da = b, df/db = a const grads = tf.grads(f); const a = tf.tensor1d([1, 2]); const b = tf.tensor1d([-1, -2]); const [da, db] = grads([a, b]); da.print(); /* Tensor [-1, -2] */ db.print(); /* Tensor [1, 2] */ |
Optimizers
tf.train.sgd
tf.train.momentum
tf.train.adagrad
tf.train.adadelta
tf.train.adam
tf.train.adamax
tf.train.rmsprop
For example, construct an Optimizer with Stochastic Gradient Descent:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
// coefficients a, b, c. const xs = tf.tensor1d([0, 1, 2, 3, 4]); const ys = tf.tensor1d([1.1, 3.9, 9.2, 15.8, 25.1]); const a = tf.scalar(Math.random()).variable(); const b = tf.scalar(Math.random()).variable(); const c = tf.scalar(Math.random()).variable(); // y = a*x^2 + b*x + c const f = x => a.mul(x.square()).add(b.mul(x)).add(c); const loss = (pred, label) => pred.sub(label).square().mean(); const LEARNING_RATE = 0.01; const optimizer = tf.train.sgd(LEARNING_RATE); // Train the model for (let i = 0; i < 10; i++) { optimizer.minimize(() => loss(f(xs), ys)); console.log(`a: ${a.dataSync()}, b: ${b.dataSync()}, c: ${c.dataSync()}`); } // Make predictions const preds = f(xs).dataSync(); preds.forEach((pred, i) => { console.log(`x: ${i}, pred: ${pred}`); }); |
Result:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
a: 2.08270001411438, b: 0.9664995670318604, c: 0.5060693621635437 a: 1.0242687463760376, b: 0.6775968670845032, c: 0.42776399850845337 a: 1.5895336866378784, b: 0.8498671650886536, c: 0.489592581987381 a: 1.2780559062957764, b: 0.7728859186172485, c: 0.4754619896411896 a: 1.4401190280914307, b: 0.8302987813949585, c: 0.5020706057548523 a: 1.346542477607727, b: 0.8149324655532837, c: 0.5064029693603516 a: 1.391097068786621, b: 0.8386674523353577, c: 0.522492527961731 a: 1.3611375093460083, b: 0.8410888314247131, c: 0.5319643020629883 a: 1.3714956045150757, b: 0.8548246026039124, c: 0.5447449684143066 a: 1.3601586818695068, b: 0.8622576594352722, c: 0.5554776191711426 x: 0, pred: 0.5554776191711426 x: 1, pred: 2.7778940200805664 x: 2, pred: 7.720627784729004 x: 3, pred: 15.383678436279297 x: 4, pred: 25.767045974731445 |
For more details, please visit: TensorFlow.js Training API
Memory Management
dispose
Calling dispose() on a tensor or variable will purge it and free up its GPU memory:
|
1 2 3 4 5 |
const x = tf.tensor2d([[1, 2], [3, 4]]); const x_squared = x.square(); x.dispose(); x_squared.dispose(); |
tf.memory
This method returns memory info object at the current time which has following properties:
– numBytes: number of bytes allocated (undisposed).
– numTensors: number of unique tensors allocated.
– numDataBuffers: number of unique data buffers allocated (undisposed).For example, tensor that is created from myTensor.reshape(newShape) shares the same data buffer with myTensor, code>numDataBuffers is alwyas ≤ the numTensors.
|
1 2 3 4 5 6 |
const myTensor = tf.tensor2d([[1, 2], [3, 4], [5, 6]], [3, 2]); const reshapedTensor = myTensor.reshape([2, 3]); console.log('numBytes: ' + tf.memory().numBytes); console.log('numTensors: ' + tf.memory().numTensors); console.log('numDataBuffers: ' + tf.memory().numDataBuffers); |
Result:
|
1 2 3 |
numBytes: 24 numTensors: 2 numDataBuffers: 1 |
tf.tidy
– executes the provided function f
– after it is executed, purges all intermediate tensors and free up their GPU memory allocated by f (except returned value by f)
*Notes:
– f should be synchronous and also not return a Promise.
– should keep code that updates the UI or makes remote requests outside of tf.tidy().
– tf.tidy() will not clean up variables. We can call dispose on them manually.
For example, a, b, and one will be cleaned up when the tidy ends.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
const result = tf.tidy(() => { // 4^1/2 + 1 const x = tf.scalar(4); const x_sqrt = x.sqrt(); const one = tf.scalar(1); console.log('numTensors (tidy): ' + tf.memory().numTensors); return x_sqrt.add(one); }); console.log('numTensors (outside tidy): ' + tf.memory().numTensors); result.print(); |
Result:
|
1 2 3 4 |
numTensors (tidy): 3 numTensors (outside tidy): 1 Tensor 3 |
tf.keep
This method keeps a tensor generated inside tf.tidy() from being disposed automatically.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
let x_sqrt; const result = tf.tidy(() => { const x = tf.scalar(4); x_sqrt = tf.keep(x.sqrt()); const one = tf.scalar(1); console.log('numTensors (tidy): ' + tf.memory().numTensors); return x_sqrt.add(one); }); console.log('numTensors (outside tidy): ' + tf.memory().numTensors); console.log('result: '); result.print(); console.log('x_sqrt: '); x_sqrt.print(); |
Result:
|
1 2 3 4 5 6 7 8 |
numTensors (tidy): 3 numTensors (outside tidy): 2 result: Tensor 3 x_sqrt: Tensor 2 |