Tests snapshots for test_transformer_lm is generate without passed into softmax layer

[fzhangtj]

According to the pdf:
Now we put the blocks together, following the high level diagram in Figure 1. Follow our description of
the embedding in Section 3.1.1, feed this into num_layers Transformer blocks, and then pass that into the
three output layers to obtain a distribution over the vocabulary

It should pass the data into softmax as last layer. But the data from snapshot is not "softmaxed", as it has negative value. My softmaxed implementation failed:

E           AssertionError: 
E           Not equal to tolerance rtol=0.0001, atol=0.0001
E           Array 'array' does not match snapshot for test_transformer_lm_truncated_input
E           Mismatched elements: 240000 / 240000 (100%)
E           Max absolute difference among violations: 12.979309
E           Max relative difference among violations: 3.9534984
E            ACTUAL: array([[[1.495662e-05, 8.760001e-03, 3.226802e-04, ..., 3.526168e-07,
E                    4.975848e-07, 1.145663e-07],
E                   [2.826909e-05, 1.762351e-02, 3.695597e-04, ..., 6.049435e-06,...
E            DESIRED: array([[[-2.785666e+00,  3.587131e+00,  2.858420e-01, ...,
E                    -6.533192e+00, -6.188807e+00, -7.657419e+00],
E                   [-2.290900e+00,  4.144320e+00,  2.796439e-01, ...,...

While the version without softmax pass the tests.
Maybe the pdf may need to point it out that we should not do softmax in last layer if the snapshot data is correct.

[SplashCloud]

I want to ask the same question too. Because it is mentioned in the manual that "... then pass that into the
three output layers", and I think the three output layers are norm, linear and softmax.