Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 84094 queued and waiting for resources
srun: job 84094 has been allocated resources
2024-10-07 07:43:07.578891: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-07 07:43:31.542342: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-07 07:44:05.393682: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-10-07 07:44:05.394470: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-10-07 07:44:05.394478: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-10-07 07:44:57.939370: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-07 07:45:00.557328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79030 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-10-07 07:45:08.506396: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 1:48:01 - loss: 2.6171 - accuracy: 0.0625
38/1563 [..............................] - ETA: 2s - loss: 2.2341 - accuracy: 0.1653
76/1563 [>.............................] - ETA: 2s - loss: 2.1727 - accuracy: 0.1941
115/1563 [=>............................] - ETA: 1s - loss: 2.1340 - accuracy: 0.2179
154/1563 [=>............................] - ETA: 1s - loss: 2.1016 - accuracy: 0.2342
193/1563 [==>...........................] - ETA: 1s - loss: 2.0767 - accuracy: 0.2463
232/1563 [===>..........................] - ETA: 1s - loss: 2.0505 - accuracy: 0.2551
271/1563 [====>.........................] - ETA: 1s - loss: 2.0298 - accuracy: 0.2621
311/1563 [====>.........................] - ETA: 1s - loss: 2.0147 - accuracy: 0.2698
350/1563 [=====>........................] - ETA: 1s - loss: 2.0001 - accuracy: 0.2727
389/1563 [======>.......................] - ETA: 1s - loss: 1.9865 - accuracy: 0.2788
428/1563 [=======>......................] - ETA: 1s - loss: 1.9744 - accuracy: 0.2837
468/1563 [=======>......................] - ETA: 1s - loss: 1.9586 - accuracy: 0.2913
507/1563 [========>.....................] - ETA: 1s - loss: 1.9483 - accuracy: 0.2967
546/1563 [=========>....................] - ETA: 1s - loss: 1.9413 - accuracy: 0.3005
586/1563 [==========>...................] - ETA: 1s - loss: 1.9297 - accuracy: 0.3058
625/1563 [==========>...................] - ETA: 1s - loss: 1.9202 - accuracy: 0.3100
665/1563 [===========>..................] - ETA: 1s - loss: 1.9142 - accuracy: 0.3136
704/1563 [============>.................] - ETA: 1s - loss: 1.9073 - accuracy: 0.3163
743/1563 [=============>................] - ETA: 1s - loss: 1.9023 - accuracy: 0.3186
782/1563 [==============>...............] - ETA: 1s - loss: 1.8958 - accuracy: 0.3214
821/1563 [==============>...............] - ETA: 0s - loss: 1.8892 - accuracy: 0.3236
860/1563 [===============>..............] - ETA: 0s - loss: 1.8823 - accuracy: 0.3252
899/1563 [================>.............] - ETA: 0s - loss: 1.8783 - accuracy: 0.3281
939/1563 [=================>............] - ETA: 0s - loss: 1.8726 - accuracy: 0.3302
979/1563 [=================>............] - ETA: 0s - loss: 1.8689 - accuracy: 0.3317
1018/1563 [==================>...........] - ETA: 0s - loss: 1.8640 - accuracy: 0.3341
1057/1563 [===================>..........] - ETA: 0s - loss: 1.8588 - accuracy: 0.3368
1096/1563 [====================>.........] - ETA: 0s - loss: 1.8540 - accuracy: 0.3388
1135/1563 [====================>.........] - ETA: 0s - loss: 1.8500 - accuracy: 0.3410
1175/1563 [=====================>........] - ETA: 0s - loss: 1.8444 - accuracy: 0.3427
1214/1563 [======================>.......] - ETA: 0s - loss: 1.8413 - accuracy: 0.3440
1253/1563 [=======================>......] - ETA: 0s - loss: 1.8385 - accuracy: 0.3455
1292/1563 [=======================>......] - ETA: 0s - loss: 1.8342 - accuracy: 0.3471
1331/1563 [========================>.....] - ETA: 0s - loss: 1.8301 - accuracy: 0.3485
1370/1563 [=========================>....] - ETA: 0s - loss: 1.8266 - accuracy: 0.3502
1409/1563 [==========================>...] - ETA: 0s - loss: 1.8224 - accuracy: 0.3516
1448/1563 [==========================>...] - ETA: 0s - loss: 1.8190 - accuracy: 0.3532
1487/1563 [===========================>..] - ETA: 0s - loss: 1.8162 - accuracy: 0.3541
1526/1563 [============================>.] - ETA: 0s - loss: 1.8134 - accuracy: 0.3554
1563/1563 [==============================] - 6s 1ms/step - loss: 1.8104 - accuracy: 0.3562
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 1.6850 - accuracy: 0.3125
40/1563 [..............................] - ETA: 1s - loss: 1.6442 - accuracy: 0.4367
79/1563 [>.............................] - ETA: 1s - loss: 1.6505 - accuracy: 0.4272
118/1563 [=>............................] - ETA: 1s - loss: 1.6454 - accuracy: 0.4253
157/1563 [==>...........................] - ETA: 1s - loss: 1.6497 - accuracy: 0.4216
196/1563 [==>...........................] - ETA: 1s - loss: 1.6429 - accuracy: 0.4241
235/1563 [===>..........................] - ETA: 1s - loss: 1.6469 - accuracy: 0.4202
274/1563 [====>.........................] - ETA: 1s - loss: 1.6443 - accuracy: 0.4215
313/1563 [=====>........................] - ETA: 1s - loss: 1.6360 - accuracy: 0.4225
352/1563 [=====>........................] - ETA: 1s - loss: 1.6373 - accuracy: 0.4222
391/1563 [======>.......................] - ETA: 1s - loss: 1.6406 - accuracy: 0.4233
430/1563 [=======>......................] - ETA: 1s - loss: 1.6407 - accuracy: 0.4229
469/1563 [========>.....................] - ETA: 1s - loss: 1.6439 - accuracy: 0.4218
508/1563 [========>.....................] - ETA: 1s - loss: 1.6456 - accuracy: 0.4212
547/1563 [=========>....................] - ETA: 1s - loss: 1.6458 - accuracy: 0.4212
587/1563 [==========>...................] - ETA: 1s - loss: 1.6427 - accuracy: 0.4234
626/1563 [===========>..................] - ETA: 1s - loss: 1.6428 - accuracy: 0.4237
665/1563 [===========>..................] - ETA: 1s - loss: 1.6387 - accuracy: 0.4259
704/1563 [============>.................] - ETA: 1s - loss: 1.6366 - accuracy: 0.4263
743/1563 [=============>................] - ETA: 1s - loss: 1.6361 - accuracy: 0.4258
782/1563 [==============>...............] - ETA: 1s - loss: 1.6358 - accuracy: 0.4258
821/1563 [==============>...............] - ETA: 0s - loss: 1.6335 - accuracy: 0.4267
860/1563 [===============>..............] - ETA: 0s - loss: 1.6329 - accuracy: 0.4261
900/1563 [================>.............] - ETA: 0s - loss: 1.6322 - accuracy: 0.4263
939/1563 [=================>............] - ETA: 0s - loss: 1.6308 - accuracy: 0.4269
978/1563 [=================>............] - ETA: 0s - loss: 1.6288 - accuracy: 0.4272
1017/1563 [==================>...........] - ETA: 0s - loss: 1.6272 - accuracy: 0.4280
1056/1563 [===================>..........] - ETA: 0s - loss: 1.6265 - accuracy: 0.4285
1094/1563 [===================>..........] - ETA: 0s - loss: 1.6270 - accuracy: 0.4290
1133/1563 [====================>.........] - ETA: 0s - loss: 1.6261 - accuracy: 0.4291
1172/1563 [=====================>........] - ETA: 0s - loss: 1.6265 - accuracy: 0.4286
1211/1563 [======================>.......] - ETA: 0s - loss: 1.6257 - accuracy: 0.4295
1250/1563 [======================>.......] - ETA: 0s - loss: 1.6251 - accuracy: 0.4295
1289/1563 [=======================>......] - ETA: 0s - loss: 1.6251 - accuracy: 0.4294
1328/1563 [========================>.....] - ETA: 0s - loss: 1.6239 - accuracy: 0.4297
1367/1563 [=========================>....] - ETA: 0s - loss: 1.6227 - accuracy: 0.4300
1406/1563 [=========================>....] - ETA: 0s - loss: 1.6221 - accuracy: 0.4300
1445/1563 [==========================>...] - ETA: 0s - loss: 1.6209 - accuracy: 0.4301
1484/1563 [===========================>..] - ETA: 0s - loss: 1.6205 - accuracy: 0.4302
1523/1563 [============================>.] - ETA: 0s - loss: 1.6202 - accuracy: 0.4298
1562/1563 [============================>.] - ETA: 0s - loss: 1.6199 - accuracy: 0.4297
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6199 - accuracy: 0.4296
Epoch 3/10
1/1563 [..............................] - ETA: 2s - loss: 1.6622 - accuracy: 0.3750
40/1563 [..............................] - ETA: 2s - loss: 1.5397 - accuracy: 0.4648
79/1563 [>.............................] - ETA: 1s - loss: 1.5459 - accuracy: 0.4644
118/1563 [=>............................] - ETA: 1s - loss: 1.5619 - accuracy: 0.4592
158/1563 [==>...........................] - ETA: 1s - loss: 1.5655 - accuracy: 0.4587
197/1563 [==>...........................] - ETA: 1s - loss: 1.5671 - accuracy: 0.4580
236/1563 [===>..........................] - ETA: 1s - loss: 1.5687 - accuracy: 0.4533
275/1563 [====>.........................] - ETA: 1s - loss: 1.5632 - accuracy: 0.4550
313/1563 [=====>........................] - ETA: 1s - loss: 1.5657 - accuracy: 0.4533
352/1563 [=====>........................] - ETA: 1s - loss: 1.5647 - accuracy: 0.4537
391/1563 [======>.......................] - ETA: 1s - loss: 1.5632 - accuracy: 0.4540
430/1563 [=======>......................] - ETA: 1s - loss: 1.5622 - accuracy: 0.4527
469/1563 [========>.....................] - ETA: 1s - loss: 1.5563 - accuracy: 0.4541
508/1563 [========>.....................] - ETA: 1s - loss: 1.5535 - accuracy: 0.4555
547/1563 [=========>....................] - ETA: 1s - loss: 1.5510 - accuracy: 0.4563
586/1563 [==========>...................] - ETA: 1s - loss: 1.5503 - accuracy: 0.4564
625/1563 [==========>...................] - ETA: 1s - loss: 1.5492 - accuracy: 0.4572
664/1563 [===========>..................] - ETA: 1s - loss: 1.5483 - accuracy: 0.4573
703/1563 [============>.................] - ETA: 1s - loss: 1.5466 - accuracy: 0.4572
742/1563 [=============>................] - ETA: 1s - loss: 1.5487 - accuracy: 0.4564
781/1563 [=============>................] - ETA: 1s - loss: 1.5469 - accuracy: 0.4563
820/1563 [==============>...............] - ETA: 0s - loss: 1.5491 - accuracy: 0.4552
859/1563 [===============>..............] - ETA: 0s - loss: 1.5489 - accuracy: 0.4560
898/1563 [================>.............] - ETA: 0s - loss: 1.5484 - accuracy: 0.4562
937/1563 [================>.............] - ETA: 0s - loss: 1.5478 - accuracy: 0.4558
976/1563 [=================>............] - ETA: 0s - loss: 1.5472 - accuracy: 0.4559
1015/1563 [==================>...........] - ETA: 0s - loss: 1.5470 - accuracy: 0.4556
1054/1563 [===================>..........] - ETA: 0s - loss: 1.5456 - accuracy: 0.4561
1093/1563 [===================>..........] - ETA: 0s - loss: 1.5451 - accuracy: 0.4568
1133/1563 [====================>.........] - ETA: 0s - loss: 1.5444 - accuracy: 0.4568
1172/1563 [=====================>........] - ETA: 0s - loss: 1.5432 - accuracy: 0.4572
1211/1563 [======================>.......] - ETA: 0s - loss: 1.5425 - accuracy: 0.4568
1250/1563 [======================>.......] - ETA: 0s - loss: 1.5435 - accuracy: 0.4567
1289/1563 [=======================>......] - ETA: 0s - loss: 1.5435 - accuracy: 0.4566
1327/1563 [========================>.....] - ETA: 0s - loss: 1.5422 - accuracy: 0.4568
1366/1563 [=========================>....] - ETA: 0s - loss: 1.5418 - accuracy: 0.4569
1405/1563 [=========================>....] - ETA: 0s - loss: 1.5408 - accuracy: 0.4574
1442/1563 [==========================>...] - ETA: 0s - loss: 1.5405 - accuracy: 0.4574
1481/1563 [===========================>..] - ETA: 0s - loss: 1.5395 - accuracy: 0.4574
1520/1563 [============================>.] - ETA: 0s - loss: 1.5399 - accuracy: 0.4574
1559/1563 [============================>.] - ETA: 0s - loss: 1.5388 - accuracy: 0.4577
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5388 - accuracy: 0.4577
Epoch 4/10
1/1563 [..............................] - ETA: 2s - loss: 1.5530 - accuracy: 0.4375
37/1563 [..............................] - ETA: 2s - loss: 1.4939 - accuracy: 0.4899
74/1563 [>.............................] - ETA: 2s - loss: 1.4765 - accuracy: 0.4865
111/1563 [=>............................] - ETA: 2s - loss: 1.4687 - accuracy: 0.4820
148/1563 [=>............................] - ETA: 1s - loss: 1.4706 - accuracy: 0.4770
185/1563 [==>...........................] - ETA: 1s - loss: 1.4698 - accuracy: 0.4794
222/1563 [===>..........................] - ETA: 1s - loss: 1.4750 - accuracy: 0.4775
259/1563 [===>..........................] - ETA: 1s - loss: 1.4826 - accuracy: 0.4749
296/1563 [====>.........................] - ETA: 1s - loss: 1.4851 - accuracy: 0.4740
333/1563 [=====>........................] - ETA: 1s - loss: 1.4943 - accuracy: 0.4699
370/1563 [======>.......................] - ETA: 1s - loss: 1.4973 - accuracy: 0.4677
407/1563 [======>.......................] - ETA: 1s - loss: 1.4960 - accuracy: 0.4682
444/1563 [=======>......................] - ETA: 1s - loss: 1.4988 - accuracy: 0.4681
481/1563 [========>.....................] - ETA: 1s - loss: 1.4981 - accuracy: 0.4681
518/1563 [========>.....................] - ETA: 1s - loss: 1.4995 - accuracy: 0.4685
555/1563 [=========>....................] - ETA: 1s - loss: 1.4960 - accuracy: 0.4711
592/1563 [==========>...................] - ETA: 1s - loss: 1.4972 - accuracy: 0.4717
628/1563 [===========>..................] - ETA: 1s - loss: 1.4951 - accuracy: 0.4722
665/1563 [===========>..................] - ETA: 1s - loss: 1.4928 - accuracy: 0.4729
702/1563 [============>.................] - ETA: 1s - loss: 1.4924 - accuracy: 0.4735
739/1563 [=============>................] - ETA: 1s - loss: 1.4917 - accuracy: 0.4737
776/1563 [=============>................] - ETA: 1s - loss: 1.4896 - accuracy: 0.4745
813/1563 [==============>...............] - ETA: 1s - loss: 1.4900 - accuracy: 0.4742
850/1563 [===============>..............] - ETA: 0s - loss: 1.4887 - accuracy: 0.4737
887/1563 [================>.............] - ETA: 0s - loss: 1.4881 - accuracy: 0.4737
924/1563 [================>.............] - ETA: 0s - loss: 1.4887 - accuracy: 0.4731
961/1563 [=================>............] - ETA: 0s - loss: 1.4864 - accuracy: 0.4741
998/1563 [==================>...........] - ETA: 0s - loss: 1.4856 - accuracy: 0.4743
1035/1563 [==================>...........] - ETA: 0s - loss: 1.4852 - accuracy: 0.4741
1072/1563 [===================>..........] - ETA: 0s - loss: 1.4852 - accuracy: 0.4740
1109/1563 [====================>.........] - ETA: 0s - loss: 1.4848 - accuracy: 0.4745
1145/1563 [====================>.........] - ETA: 0s - loss: 1.4850 - accuracy: 0.4740
1182/1563 [=====================>........] - ETA: 0s - loss: 1.4850 - accuracy: 0.4738
1219/1563 [======================>.......] - ETA: 0s - loss: 1.4843 - accuracy: 0.4746
1257/1563 [=======================>......] - ETA: 0s - loss: 1.4831 - accuracy: 0.4752
1296/1563 [=======================>......] - ETA: 0s - loss: 1.4822 - accuracy: 0.4754
1335/1563 [========================>.....] - ETA: 0s - loss: 1.4814 - accuracy: 0.4762
1374/1563 [=========================>....] - ETA: 0s - loss: 1.4794 - accuracy: 0.4774
1413/1563 [==========================>...] - ETA: 0s - loss: 1.4792 - accuracy: 0.4773
1452/1563 [==========================>...] - ETA: 0s - loss: 1.4803 - accuracy: 0.4773
1491/1563 [===========================>..] - ETA: 0s - loss: 1.4800 - accuracy: 0.4777
1530/1563 [============================>.] - ETA: 0s - loss: 1.4814 - accuracy: 0.4768
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4810 - accuracy: 0.4771
Epoch 5/10
1/1563 [..............................] - ETA: 2s - loss: 1.2259 - accuracy: 0.5312
40/1563 [..............................] - ETA: 2s - loss: 1.4304 - accuracy: 0.4852
79/1563 [>.............................] - ETA: 1s - loss: 1.4179 - accuracy: 0.4834
118/1563 [=>............................] - ETA: 1s - loss: 1.4218 - accuracy: 0.4915
157/1563 [==>...........................] - ETA: 1s - loss: 1.4149 - accuracy: 0.4974
196/1563 [==>...........................] - ETA: 1s - loss: 1.4232 - accuracy: 0.4954
235/1563 [===>..........................] - ETA: 1s - loss: 1.4226 - accuracy: 0.4961
274/1563 [====>.........................] - ETA: 1s - loss: 1.4244 - accuracy: 0.4951
311/1563 [====>.........................] - ETA: 1s - loss: 1.4328 - accuracy: 0.4940
348/1563 [=====>........................] - ETA: 1s - loss: 1.4371 - accuracy: 0.4943
385/1563 [======>.......................] - ETA: 1s - loss: 1.4381 - accuracy: 0.4927
422/1563 [=======>......................] - ETA: 1s - loss: 1.4392 - accuracy: 0.4925
459/1563 [=======>......................] - ETA: 1s - loss: 1.4385 - accuracy: 0.4921
496/1563 [========>.....................] - ETA: 1s - loss: 1.4374 - accuracy: 0.4928
533/1563 [=========>....................] - ETA: 1s - loss: 1.4421 - accuracy: 0.4909
570/1563 [=========>....................] - ETA: 1s - loss: 1.4443 - accuracy: 0.4900
608/1563 [==========>...................] - ETA: 1s - loss: 1.4430 - accuracy: 0.4905
647/1563 [===========>..................] - ETA: 1s - loss: 1.4399 - accuracy: 0.4912
686/1563 [============>.................] - ETA: 1s - loss: 1.4425 - accuracy: 0.4912
725/1563 [============>.................] - ETA: 1s - loss: 1.4431 - accuracy: 0.4910
764/1563 [=============>................] - ETA: 1s - loss: 1.4425 - accuracy: 0.4910
803/1563 [==============>...............] - ETA: 1s - loss: 1.4427 - accuracy: 0.4918
842/1563 [===============>..............] - ETA: 0s - loss: 1.4424 - accuracy: 0.4909
881/1563 [===============>..............] - ETA: 0s - loss: 1.4417 - accuracy: 0.4910
919/1563 [================>.............] - ETA: 0s - loss: 1.4417 - accuracy: 0.4913
957/1563 [=================>............] - ETA: 0s - loss: 1.4417 - accuracy: 0.4920
995/1563 [==================>...........] - ETA: 0s - loss: 1.4418 - accuracy: 0.4919
1033/1563 [==================>...........] - ETA: 0s - loss: 1.4399 - accuracy: 0.4930
1072/1563 [===================>..........] - ETA: 0s - loss: 1.4373 - accuracy: 0.4944
1110/1563 [====================>.........] - ETA: 0s - loss: 1.4361 - accuracy: 0.4949
1149/1563 [=====================>........] - ETA: 0s - loss: 1.4366 - accuracy: 0.4948
1188/1563 [=====================>........] - ETA: 0s - loss: 1.4358 - accuracy: 0.4948
1227/1563 [======================>.......] - ETA: 0s - loss: 1.4363 - accuracy: 0.4950
1265/1563 [=======================>......] - ETA: 0s - loss: 1.4352 - accuracy: 0.4958
1303/1563 [========================>.....] - ETA: 0s - loss: 1.4335 - accuracy: 0.4966
1342/1563 [========================>.....] - ETA: 0s - loss: 1.4322 - accuracy: 0.4969
1380/1563 [=========================>....] - ETA: 0s - loss: 1.4321 - accuracy: 0.4967
1418/1563 [==========================>...] - ETA: 0s - loss: 1.4329 - accuracy: 0.4965
1457/1563 [==========================>...] - ETA: 0s - loss: 1.4322 - accuracy: 0.4968
1496/1563 [===========================>..] - ETA: 0s - loss: 1.4305 - accuracy: 0.4971
1534/1563 [============================>.] - ETA: 0s - loss: 1.4301 - accuracy: 0.4970
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4295 - accuracy: 0.4973
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.8597 - accuracy: 0.4062
39/1563 [..............................] - ETA: 2s - loss: 1.3857 - accuracy: 0.5256
77/1563 [>.............................] - ETA: 1s - loss: 1.3848 - accuracy: 0.5223
115/1563 [=>............................] - ETA: 1s - loss: 1.3810 - accuracy: 0.5228
154/1563 [=>............................] - ETA: 1s - loss: 1.3822 - accuracy: 0.5179
193/1563 [==>...........................] - ETA: 1s - loss: 1.3834 - accuracy: 0.5168
232/1563 [===>..........................] - ETA: 1s - loss: 1.3844 - accuracy: 0.5135
270/1563 [====>.........................] - ETA: 1s - loss: 1.3895 - accuracy: 0.5119
308/1563 [====>.........................] - ETA: 1s - loss: 1.3807 - accuracy: 0.5172
346/1563 [=====>........................] - ETA: 1s - loss: 1.3822 - accuracy: 0.5174
385/1563 [======>.......................] - ETA: 1s - loss: 1.3810 - accuracy: 0.5172
423/1563 [=======>......................] - ETA: 1s - loss: 1.3803 - accuracy: 0.5168
462/1563 [=======>......................] - ETA: 1s - loss: 1.3831 - accuracy: 0.5147
501/1563 [========>.....................] - ETA: 1s - loss: 1.3795 - accuracy: 0.5155
540/1563 [=========>....................] - ETA: 1s - loss: 1.3802 - accuracy: 0.5153
578/1563 [==========>...................] - ETA: 1s - loss: 1.3842 - accuracy: 0.5146
617/1563 [==========>...................] - ETA: 1s - loss: 1.3857 - accuracy: 0.5142
655/1563 [===========>..................] - ETA: 1s - loss: 1.3868 - accuracy: 0.5136
693/1563 [============>.................] - ETA: 1s - loss: 1.3871 - accuracy: 0.5133
731/1563 [=============>................] - ETA: 1s - loss: 1.3880 - accuracy: 0.5133
770/1563 [=============>................] - ETA: 1s - loss: 1.3889 - accuracy: 0.5134
808/1563 [==============>...............] - ETA: 0s - loss: 1.3894 - accuracy: 0.5132
847/1563 [===============>..............] - ETA: 0s - loss: 1.3885 - accuracy: 0.5131
886/1563 [================>.............] - ETA: 0s - loss: 1.3896 - accuracy: 0.5128
924/1563 [================>.............] - ETA: 0s - loss: 1.3903 - accuracy: 0.5123
962/1563 [=================>............] - ETA: 0s - loss: 1.3905 - accuracy: 0.5123
1000/1563 [==================>...........] - ETA: 0s - loss: 1.3901 - accuracy: 0.5119
1039/1563 [==================>...........] - ETA: 0s - loss: 1.3896 - accuracy: 0.5118
1078/1563 [===================>..........] - ETA: 0s - loss: 1.3903 - accuracy: 0.5115
1116/1563 [====================>.........] - ETA: 0s - loss: 1.3916 - accuracy: 0.5112
1155/1563 [=====================>........] - ETA: 0s - loss: 1.3919 - accuracy: 0.5108
1194/1563 [=====================>........] - ETA: 0s - loss: 1.3909 - accuracy: 0.5113
1233/1563 [======================>.......] - ETA: 0s - loss: 1.3921 - accuracy: 0.5110
1272/1563 [=======================>......] - ETA: 0s - loss: 1.3905 - accuracy: 0.5115
1311/1563 [========================>.....] - ETA: 0s - loss: 1.3891 - accuracy: 0.5123
1349/1563 [========================>.....] - ETA: 0s - loss: 1.3880 - accuracy: 0.5127
1388/1563 [=========================>....] - ETA: 0s - loss: 1.3885 - accuracy: 0.5123
1427/1563 [==========================>...] - ETA: 0s - loss: 1.3889 - accuracy: 0.5120
1466/1563 [===========================>..] - ETA: 0s - loss: 1.3880 - accuracy: 0.5125
1504/1563 [===========================>..] - ETA: 0s - loss: 1.3884 - accuracy: 0.5122
1542/1563 [============================>.] - ETA: 0s - loss: 1.3878 - accuracy: 0.5123
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3864 - accuracy: 0.5127
Epoch 7/10
1/1563 [..............................] - ETA: 1s - loss: 1.6256 - accuracy: 0.5000
38/1563 [..............................] - ETA: 2s - loss: 1.3111 - accuracy: 0.5452
77/1563 [>.............................] - ETA: 1s - loss: 1.3418 - accuracy: 0.5353
116/1563 [=>............................] - ETA: 1s - loss: 1.3553 - accuracy: 0.5202
155/1563 [=>............................] - ETA: 1s - loss: 1.3563 - accuracy: 0.5212
194/1563 [==>...........................] - ETA: 1s - loss: 1.3492 - accuracy: 0.5238
233/1563 [===>..........................] - ETA: 1s - loss: 1.3472 - accuracy: 0.5271
272/1563 [====>.........................] - ETA: 1s - loss: 1.3507 - accuracy: 0.5262
310/1563 [====>.........................] - ETA: 1s - loss: 1.3489 - accuracy: 0.5260
348/1563 [=====>........................] - ETA: 1s - loss: 1.3503 - accuracy: 0.5250
387/1563 [======>.......................] - ETA: 1s - loss: 1.3445 - accuracy: 0.5285
426/1563 [=======>......................] - ETA: 1s - loss: 1.3409 - accuracy: 0.5303
465/1563 [=======>......................] - ETA: 1s - loss: 1.3411 - accuracy: 0.5288
504/1563 [========>.....................] - ETA: 1s - loss: 1.3431 - accuracy: 0.5278
543/1563 [=========>....................] - ETA: 1s - loss: 1.3420 - accuracy: 0.5288
582/1563 [==========>...................] - ETA: 1s - loss: 1.3424 - accuracy: 0.5298
621/1563 [==========>...................] - ETA: 1s - loss: 1.3436 - accuracy: 0.5289
659/1563 [===========>..................] - ETA: 1s - loss: 1.3414 - accuracy: 0.5302
697/1563 [============>.................] - ETA: 1s - loss: 1.3432 - accuracy: 0.5308
735/1563 [=============>................] - ETA: 1s - loss: 1.3442 - accuracy: 0.5294
774/1563 [=============>................] - ETA: 1s - loss: 1.3442 - accuracy: 0.5295
812/1563 [==============>...............] - ETA: 0s - loss: 1.3438 - accuracy: 0.5301
851/1563 [===============>..............] - ETA: 0s - loss: 1.3423 - accuracy: 0.5305
889/1563 [================>.............] - ETA: 0s - loss: 1.3430 - accuracy: 0.5295
927/1563 [================>.............] - ETA: 0s - loss: 1.3450 - accuracy: 0.5295
965/1563 [=================>............] - ETA: 0s - loss: 1.3437 - accuracy: 0.5298
1004/1563 [==================>...........] - ETA: 0s - loss: 1.3431 - accuracy: 0.5300
1043/1563 [===================>..........] - ETA: 0s - loss: 1.3444 - accuracy: 0.5297
1081/1563 [===================>..........] - ETA: 0s - loss: 1.3441 - accuracy: 0.5305
1119/1563 [====================>.........] - ETA: 0s - loss: 1.3457 - accuracy: 0.5297
1157/1563 [=====================>........] - ETA: 0s - loss: 1.3457 - accuracy: 0.5299
1195/1563 [=====================>........] - ETA: 0s - loss: 1.3459 - accuracy: 0.5297
1233/1563 [======================>.......] - ETA: 0s - loss: 1.3464 - accuracy: 0.5292
1271/1563 [=======================>......] - ETA: 0s - loss: 1.3472 - accuracy: 0.5291
1309/1563 [========================>.....] - ETA: 0s - loss: 1.3478 - accuracy: 0.5291
1348/1563 [========================>.....] - ETA: 0s - loss: 1.3482 - accuracy: 0.5293
1387/1563 [=========================>....] - ETA: 0s - loss: 1.3476 - accuracy: 0.5292
1426/1563 [==========================>...] - ETA: 0s - loss: 1.3479 - accuracy: 0.5289
1465/1563 [===========================>..] - ETA: 0s - loss: 1.3493 - accuracy: 0.5282
1503/1563 [===========================>..] - ETA: 0s - loss: 1.3504 - accuracy: 0.5278
1542/1563 [============================>.] - ETA: 0s - loss: 1.3506 - accuracy: 0.5273
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3509 - accuracy: 0.5273
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.5362 - accuracy: 0.3750
39/1563 [..............................] - ETA: 2s - loss: 1.3406 - accuracy: 0.5321
78/1563 [>.............................] - ETA: 1s - loss: 1.3271 - accuracy: 0.5369
117/1563 [=>............................] - ETA: 1s - loss: 1.3129 - accuracy: 0.5419
156/1563 [=>............................] - ETA: 1s - loss: 1.3226 - accuracy: 0.5367
194/1563 [==>...........................] - ETA: 1s - loss: 1.3202 - accuracy: 0.5403
232/1563 [===>..........................] - ETA: 1s - loss: 1.3169 - accuracy: 0.5408
270/1563 [====>.........................] - ETA: 1s - loss: 1.3143 - accuracy: 0.5413
309/1563 [====>.........................] - ETA: 1s - loss: 1.3136 - accuracy: 0.5395
347/1563 [=====>........................] - ETA: 1s - loss: 1.3104 - accuracy: 0.5427
386/1563 [======>.......................] - ETA: 1s - loss: 1.3101 - accuracy: 0.5423
425/1563 [=======>......................] - ETA: 1s - loss: 1.3098 - accuracy: 0.5421
464/1563 [=======>......................] - ETA: 1s - loss: 1.3081 - accuracy: 0.5434
503/1563 [========>.....................] - ETA: 1s - loss: 1.3133 - accuracy: 0.5407
542/1563 [=========>....................] - ETA: 1s - loss: 1.3150 - accuracy: 0.5406
580/1563 [==========>...................] - ETA: 1s - loss: 1.3163 - accuracy: 0.5398
618/1563 [==========>...................] - ETA: 1s - loss: 1.3152 - accuracy: 0.5405
657/1563 [===========>..................] - ETA: 1s - loss: 1.3180 - accuracy: 0.5396
696/1563 [============>.................] - ETA: 1s - loss: 1.3184 - accuracy: 0.5396
735/1563 [=============>................] - ETA: 1s - loss: 1.3197 - accuracy: 0.5390
774/1563 [=============>................] - ETA: 1s - loss: 1.3196 - accuracy: 0.5384
812/1563 [==============>...............] - ETA: 0s - loss: 1.3217 - accuracy: 0.5370
851/1563 [===============>..............] - ETA: 0s - loss: 1.3197 - accuracy: 0.5381
889/1563 [================>.............] - ETA: 0s - loss: 1.3208 - accuracy: 0.5374
927/1563 [================>.............] - ETA: 0s - loss: 1.3208 - accuracy: 0.5376
965/1563 [=================>............] - ETA: 0s - loss: 1.3200 - accuracy: 0.5375
1004/1563 [==================>...........] - ETA: 0s - loss: 1.3186 - accuracy: 0.5383
1043/1563 [===================>..........] - ETA: 0s - loss: 1.3182 - accuracy: 0.5382
1082/1563 [===================>..........] - ETA: 0s - loss: 1.3190 - accuracy: 0.5379
1120/1563 [====================>.........] - ETA: 0s - loss: 1.3200 - accuracy: 0.5376
1158/1563 [=====================>........] - ETA: 0s - loss: 1.3195 - accuracy: 0.5382
1196/1563 [=====================>........] - ETA: 0s - loss: 1.3188 - accuracy: 0.5387
1234/1563 [======================>.......] - ETA: 0s - loss: 1.3161 - accuracy: 0.5395
1273/1563 [=======================>......] - ETA: 0s - loss: 1.3161 - accuracy: 0.5394
1312/1563 [========================>.....] - ETA: 0s - loss: 1.3160 - accuracy: 0.5389
1351/1563 [========================>.....] - ETA: 0s - loss: 1.3165 - accuracy: 0.5385
1390/1563 [=========================>....] - ETA: 0s - loss: 1.3162 - accuracy: 0.5386
1428/1563 [==========================>...] - ETA: 0s - loss: 1.3161 - accuracy: 0.5385
1467/1563 [===========================>..] - ETA: 0s - loss: 1.3166 - accuracy: 0.5384
1506/1563 [===========================>..] - ETA: 0s - loss: 1.3167 - accuracy: 0.5383
1544/1563 [============================>.] - ETA: 0s - loss: 1.3164 - accuracy: 0.5388
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3163 - accuracy: 0.5390
Epoch 9/10
1/1563 [..............................] - ETA: 2s - loss: 1.4140 - accuracy: 0.5625
39/1563 [..............................] - ETA: 2s - loss: 1.3269 - accuracy: 0.5489
77/1563 [>.............................] - ETA: 1s - loss: 1.3022 - accuracy: 0.5556
116/1563 [=>............................] - ETA: 1s - loss: 1.2908 - accuracy: 0.5552
154/1563 [=>............................] - ETA: 1s - loss: 1.2696 - accuracy: 0.5623
192/1563 [==>...........................] - ETA: 1s - loss: 1.2804 - accuracy: 0.5535
230/1563 [===>..........................] - ETA: 1s - loss: 1.2791 - accuracy: 0.5534
268/1563 [====>.........................] - ETA: 1s - loss: 1.2857 - accuracy: 0.5491
306/1563 [====>.........................] - ETA: 1s - loss: 1.2785 - accuracy: 0.5520
344/1563 [=====>........................] - ETA: 1s - loss: 1.2840 - accuracy: 0.5505
383/1563 [======>.......................] - ETA: 1s - loss: 1.2847 - accuracy: 0.5519
421/1563 [=======>......................] - ETA: 1s - loss: 1.2881 - accuracy: 0.5509
459/1563 [=======>......................] - ETA: 1s - loss: 1.2880 - accuracy: 0.5500
497/1563 [========>.....................] - ETA: 1s - loss: 1.2887 - accuracy: 0.5494
535/1563 [=========>....................] - ETA: 1s - loss: 1.2905 - accuracy: 0.5481
573/1563 [=========>....................] - ETA: 1s - loss: 1.2884 - accuracy: 0.5500
612/1563 [==========>...................] - ETA: 1s - loss: 1.2850 - accuracy: 0.5510
650/1563 [===========>..................] - ETA: 1s - loss: 1.2855 - accuracy: 0.5518
689/1563 [============>.................] - ETA: 1s - loss: 1.2857 - accuracy: 0.5513
728/1563 [============>.................] - ETA: 1s - loss: 1.2874 - accuracy: 0.5511
767/1563 [=============>................] - ETA: 1s - loss: 1.2891 - accuracy: 0.5505
806/1563 [==============>...............] - ETA: 0s - loss: 1.2891 - accuracy: 0.5511
844/1563 [===============>..............] - ETA: 0s - loss: 1.2889 - accuracy: 0.5507
882/1563 [===============>..............] - ETA: 0s - loss: 1.2889 - accuracy: 0.5511
920/1563 [================>.............] - ETA: 0s - loss: 1.2890 - accuracy: 0.5505
959/1563 [=================>............] - ETA: 0s - loss: 1.2882 - accuracy: 0.5506
998/1563 [==================>...........] - ETA: 0s - loss: 1.2891 - accuracy: 0.5497
1037/1563 [==================>...........] - ETA: 0s - loss: 1.2860 - accuracy: 0.5504
1076/1563 [===================>..........] - ETA: 0s - loss: 1.2871 - accuracy: 0.5504
1115/1563 [====================>.........] - ETA: 0s - loss: 1.2867 - accuracy: 0.5508
1154/1563 [=====================>........] - ETA: 0s - loss: 1.2875 - accuracy: 0.5507
1192/1563 [=====================>........] - ETA: 0s - loss: 1.2857 - accuracy: 0.5512
1231/1563 [======================>.......] - ETA: 0s - loss: 1.2857 - accuracy: 0.5508
1270/1563 [=======================>......] - ETA: 0s - loss: 1.2854 - accuracy: 0.5510
1309/1563 [========================>.....] - ETA: 0s - loss: 1.2856 - accuracy: 0.5505
1348/1563 [========================>.....] - ETA: 0s - loss: 1.2849 - accuracy: 0.5503
1387/1563 [=========================>....] - ETA: 0s - loss: 1.2844 - accuracy: 0.5509
1425/1563 [==========================>...] - ETA: 0s - loss: 1.2832 - accuracy: 0.5508
1464/1563 [===========================>..] - ETA: 0s - loss: 1.2824 - accuracy: 0.5509
1503/1563 [===========================>..] - ETA: 0s - loss: 1.2834 - accuracy: 0.5508
1542/1563 [============================>.] - ETA: 0s - loss: 1.2837 - accuracy: 0.5507
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2839 - accuracy: 0.5505
Epoch 10/10
1/1563 [..............................] - ETA: 1s - loss: 1.2584 - accuracy: 0.5625
40/1563 [..............................] - ETA: 1s - loss: 1.2364 - accuracy: 0.5695
79/1563 [>.............................] - ETA: 1s - loss: 1.2515 - accuracy: 0.5633
118/1563 [=>............................] - ETA: 1s - loss: 1.2647 - accuracy: 0.5546
156/1563 [=>............................] - ETA: 1s - loss: 1.2584 - accuracy: 0.5579
195/1563 [==>...........................] - ETA: 1s - loss: 1.2587 - accuracy: 0.5604
234/1563 [===>..........................] - ETA: 1s - loss: 1.2556 - accuracy: 0.5590
272/1563 [====>.........................] - ETA: 1s - loss: 1.2563 - accuracy: 0.5600
311/1563 [====>.........................] - ETA: 1s - loss: 1.2504 - accuracy: 0.5612
349/1563 [=====>........................] - ETA: 1s - loss: 1.2507 - accuracy: 0.5599
387/1563 [======>.......................] - ETA: 1s - loss: 1.2522 - accuracy: 0.5577
426/1563 [=======>......................] - ETA: 1s - loss: 1.2528 - accuracy: 0.5564
465/1563 [=======>......................] - ETA: 1s - loss: 1.2530 - accuracy: 0.5571
503/1563 [========>.....................] - ETA: 1s - loss: 1.2518 - accuracy: 0.5589
542/1563 [=========>....................] - ETA: 1s - loss: 1.2575 - accuracy: 0.5571
580/1563 [==========>...................] - ETA: 1s - loss: 1.2578 - accuracy: 0.5577
617/1563 [==========>...................] - ETA: 1s - loss: 1.2602 - accuracy: 0.5569
655/1563 [===========>..................] - ETA: 1s - loss: 1.2597 - accuracy: 0.5580
693/1563 [============>.................] - ETA: 1s - loss: 1.2578 - accuracy: 0.5580
731/1563 [=============>................] - ETA: 1s - loss: 1.2600 - accuracy: 0.5572
769/1563 [=============>................] - ETA: 1s - loss: 1.2581 - accuracy: 0.5588
807/1563 [==============>...............] - ETA: 0s - loss: 1.2563 - accuracy: 0.5594
845/1563 [===============>..............] - ETA: 0s - loss: 1.2573 - accuracy: 0.5595
884/1563 [===============>..............] - ETA: 0s - loss: 1.2581 - accuracy: 0.5595
922/1563 [================>.............] - ETA: 0s - loss: 1.2563 - accuracy: 0.5602
960/1563 [=================>............] - ETA: 0s - loss: 1.2568 - accuracy: 0.5600
998/1563 [==================>...........] - ETA: 0s - loss: 1.2559 - accuracy: 0.5603
1036/1563 [==================>...........] - ETA: 0s - loss: 1.2554 - accuracy: 0.5603
1075/1563 [===================>..........] - ETA: 0s - loss: 1.2545 - accuracy: 0.5607
1114/1563 [====================>.........] - ETA: 0s - loss: 1.2552 - accuracy: 0.5607
1152/1563 [=====================>........] - ETA: 0s - loss: 1.2546 - accuracy: 0.5610
1191/1563 [=====================>........] - ETA: 0s - loss: 1.2554 - accuracy: 0.5606
1229/1563 [======================>.......] - ETA: 0s - loss: 1.2558 - accuracy: 0.5605
1267/1563 [=======================>......] - ETA: 0s - loss: 1.2568 - accuracy: 0.5602
1306/1563 [========================>.....] - ETA: 0s - loss: 1.2557 - accuracy: 0.5605
1345/1563 [========================>.....] - ETA: 0s - loss: 1.2550 - accuracy: 0.5611
1384/1563 [=========================>....] - ETA: 0s - loss: 1.2540 - accuracy: 0.5618
1423/1563 [==========================>...] - ETA: 0s - loss: 1.2540 - accuracy: 0.5617
1462/1563 [===========================>..] - ETA: 0s - loss: 1.2536 - accuracy: 0.5621
1501/1563 [===========================>..] - ETA: 0s - loss: 1.2532 - accuracy: 0.5622
1539/1563 [============================>.] - ETA: 0s - loss: 1.2531 - accuracy: 0.5620
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2517 - accuracy: 0.5624
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 37.23993635177612
hydro05
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins722171026689368246.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
