Gradient descent calculation split in 2 steps
In a machine learning course I am taking we have an assignment with a notebook (In linear models). So far I have calculated the cost and the sigmoid in a logistic regression and the next exercise wants the gradient. I am pretty aware of the formula and how it works theoretically,but heres what got me confused : The function is split functions,the first one wants to calculate the derivatives of weights,bias and the 2nd to update them accordingly. My problem is this,I dont understand how by the GD formula I can "extract" the derivatives I need since in my understanding,it could be done simply in a loop in one function. I'll leave the code below in case anyone wants to take a look at it but please dont answer with code,i just want the theoretical direction. Thank you
First method
def gradient(w,b,X,Y,sigma): """ Arguments: w -- weights b -- bias X -- input data Y -- target or label vector Return: dw -- gradient of the loss with respect to w (numpy array) db -- gradient of the loss with respect to b (scalar) """ # BEGIN CODE HERE bw = ... #(Optional) X = ... #(Optional) grad = ... #(Optional) dw = ... db = ... #END CODE HERE return dw, dbSecond method
def update_parameters(w,b,X,Y,num_iterations,learning_rate): """ This function optimizes w and b by running a gradient descent algorithm Arguments: w -- weights b -- bias X -- input data Y -- target or label vector num_iterations -- number of iterations of the optimization loop learning_rate -- learning rate of the gradient descent update rule Returns: params -- dictionary containing the weights w and bias b grads -- dictionary containing the gradients of the weights and bias with respect to the cost function. """ for i in range(num_iterations): w_prev = w b_prev = b # BEGIN CODE HERE # Cost and gradient calculation sigma, cost = ... dw, db = ... # update rule w = w_prev - ... b = b_prev - ... #END CODE HERE # Print the cost every 100 training iterations if i % 100 == 0: print ("Cost after iteration %i: %f" %(i, cost)) return w,b,dw,db
