In the beginning, you want artificial knowledge to work with. The info ought to present some nonlinear dependence. Let’s outline it like this:
In Python, it has the next kind:
np.random.seed(42)
X = np.random.regular(1, 4.5, 10000)
y = np.piecewise(X, [X < -2,(X >= -2) & (X < 2), X >= 2], [lambda X: 2*X + 5, lambda X: 7.3*np.sin(X), lambda X: -0.03*X**3 + 2]) + np.random.regular(0, 1, X.form)
After visualization:
Since we’re visualizing a 3D house, the neural community solely has two weights. Because of this the ANN consists of a single hidden neuron. Implementing this in PyTorch may be very intuitive.
class ANN(nn.Module):
def __init__(self, input_size, N, output_size):
tremendous().__init__()
self.internet = nn.Sequential()
self.internet.add_module(title='Layer_1', module=nn.Linear(input_size, N, bias=False))
self.internet.add_module(title='Tanh',module=nn.Tanh())
self.internet.add_module(title='Layer_2',module=nn.Linear(N, output_size, bias=False))def ahead(self, x):
return self.internet(x)
necessary! Do not forget to show off layer bias. In case you do not try this, ×2 Much more parameters.
To assemble the error floor, we first have to create a grid of potential values for W1 and W2. Then, for every mixture of weights, replace the community parameters and calculate the error.
W1, W2 = np.arange(-2, 2, 0.05), np.arange(-2, 2, 0.05)
LOSS = np.zeros((len(W1), len(W2)))
for i, w1 in enumerate(W1):
mannequin.internet._modules['Layer_1'].weight.knowledge = torch.tensor([[w1]], dtype=torch.float32)for j, w2 in enumerate(W2):
mannequin.internet._modules['Layer_2'].weight.knowledge = torch.tensor([[w2]], dtype=torch.float32)
mannequin.eval()
total_loss = 0
with torch.no_grad():
for x, y in test_loader:
preds = mannequin(x.reshape(-1, 1))
total_loss += loss(preds, y).merchandise()
LOSS[i, j] = total_loss / len(test_loader)
It could take a while. If the decision of this grid (i.e., the step measurement between potential weight values) is just too coarse, native minima and maxima will be missed. Bear in mind how studying charges are sometimes scheduled to lower over time? While you do that, absolutely the change in weight values will be as small as 1e-3 or much less . A grid of 0.5 steps can’t seize any of those effective particulars of the error floor.
At this level, we do not care concerning the high quality of the educated mannequin in any respect. Nonetheless, we need to watch out concerning the studying charge, so we maintain it between 1e-1 and 1e-2. Simply gather the burden values and errors throughout the coaching course of and save them in a separate listing.
mannequin = ANN(1,1,1)
epochs = 25
lr = 1e-2optimizer = optim.SGD(mannequin.parameters(),lr =lr)
mannequin.internet._modules['Layer_1'].weight.knowledge = torch.tensor([[-1]], dtype=torch.float32)
mannequin.internet._modules['Layer_2'].weight.knowledge = torch.tensor([[-1]], dtype=torch.float32)
errors, weights_1, weights_2 = [], [], []
mannequin.eval()
with torch.no_grad():
total_loss = 0
for x, y in test_loader:
preds = mannequin(x.reshape(-1,1))
error = loss(preds, y)
total_loss += error.merchandise()
weights_1.append(mannequin.internet._modules['Layer_1'].weight.knowledge.merchandise())
weights_2.append(mannequin.internet._modules['Layer_2'].weight.knowledge.merchandise())
errors.append(total_loss / len(test_loader))
for epoch in tqdm(vary(epochs)):
mannequin.practice()
for x, y in train_loader:
pred = mannequin(x.reshape(-1,1))
error = loss(pred, y)
optimizer.zero_grad()
error.backward()
optimizer.step()
mannequin.eval()
test_preds, true = [], []
with torch.no_grad():
total_loss = 0
for x, y in test_loader:
preds = mannequin(x.reshape(-1,1))
error = loss(preds, y)
test_preds.append(preds)
true.append(y)
total_loss += error.merchandise()
weights_1.append(mannequin.internet._modules['Layer_1'].weight.knowledge.merchandise())
weights_2.append(mannequin.internet._modules['Layer_2'].weight.knowledge.merchandise())
errors.append(total_loss / len(test_loader))
Lastly, you’ll be able to visualize the collected knowledge utilizing plotly. The plot has two scenes: the floor and the SGD orbit. One method to accomplish the primary half is to create a determine utilizing a plot. floor. Then replace the structure and elegance it a bit.
The second half may be very easy. Simply use it. scatter 3d Specify all three axes utilizing capabilities.
import plotly.graph_objects as go
import plotly.io as pioplotly_template = pio.templates["plotly_dark"]
fig = go.Determine(knowledge=[go.Surface(z=LOSS, x=W1, y=W2)])
fig.update_layout(
title='Loss Floor',
scene=dict(
xaxis_title='w1',
yaxis_title='w2',
zaxis_title='Loss',
aspectmode='guide',
aspectratio=dict(x=1, y=1, z=0.5),
xaxis=dict(showgrid=False),
yaxis=dict(showgrid=False),
zaxis=dict(showgrid=False),
),
width=800,
top=800
)
fig.add_trace(go.Scatter3d(x=weights_2, y=weights_1, z=errors,
mode='strains+markers',
line=dict(colour='purple', width=2),
marker=dict(measurement=4, colour='yellow') ))
fig.present()
Run regionally in Google Colab or Jupyter Pocket book to higher examine the error floor. To be trustworthy, I spent lots of time simply this image 🙂
I would like to see the floor, so be happy to share within the feedback. I strongly consider that the extra imperfect a floor is, the extra fascinating it’s to analyze.
===========================================
All of my publications on Medium are free and open entry. So I might actually admire it for those who may observe me right here.
Ps I am very obsessed with (geo) knowledge science, ML/AI, and local weather change. If you want to work collectively on a mission please contact me. linkedin and take a look at my website!
🛰️ Comply with us for extra particulars 🛰️
