Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 97048 queued and waiting for resources
srun: job 97048 has been allocated resources
2024-11-10 07:43:02.850693: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-10 07:43:04.074766: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-10 07:43:07.306482: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-10 07:43:07.307723: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-10 07:43:07.307731: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-11-10 07:43:14.763103: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-10 07:43:16.151815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-11-10 07:43:20.249181: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 1:16:17 - loss: 2.2973 - accuracy: 0.1250
36/1563 [..............................] - ETA: 2s - loss: 2.2387 - accuracy: 0.1545
73/1563 [>.............................] - ETA: 2s - loss: 2.1689 - accuracy: 0.1871
110/1563 [=>............................] - ETA: 2s - loss: 2.1153 - accuracy: 0.2165
147/1563 [=>............................] - ETA: 1s - loss: 2.0843 - accuracy: 0.2343
184/1563 [==>...........................] - ETA: 1s - loss: 2.0576 - accuracy: 0.2493
222/1563 [===>..........................] - ETA: 1s - loss: 2.0363 - accuracy: 0.2608
260/1563 [===>..........................] - ETA: 1s - loss: 2.0155 - accuracy: 0.2691
298/1563 [====>.........................] - ETA: 1s - loss: 1.9965 - accuracy: 0.2760
336/1563 [=====>........................] - ETA: 1s - loss: 1.9816 - accuracy: 0.2823
374/1563 [======>.......................] - ETA: 1s - loss: 1.9670 - accuracy: 0.2894
411/1563 [======>.......................] - ETA: 1s - loss: 1.9580 - accuracy: 0.2942
449/1563 [=======>......................] - ETA: 1s - loss: 1.9499 - accuracy: 0.2983
487/1563 [========>.....................] - ETA: 1s - loss: 1.9422 - accuracy: 0.3012
525/1563 [=========>....................] - ETA: 1s - loss: 1.9365 - accuracy: 0.3035
563/1563 [=========>....................] - ETA: 1s - loss: 1.9296 - accuracy: 0.3055
601/1563 [==========>...................] - ETA: 1s - loss: 1.9234 - accuracy: 0.3074
638/1563 [===========>..................] - ETA: 1s - loss: 1.9158 - accuracy: 0.3099
676/1563 [===========>..................] - ETA: 1s - loss: 1.9064 - accuracy: 0.3138
713/1563 [============>.................] - ETA: 1s - loss: 1.8985 - accuracy: 0.3171
750/1563 [=============>................] - ETA: 1s - loss: 1.8940 - accuracy: 0.3198
788/1563 [==============>...............] - ETA: 1s - loss: 1.8897 - accuracy: 0.3214
826/1563 [==============>...............] - ETA: 0s - loss: 1.8838 - accuracy: 0.3237
864/1563 [===============>..............] - ETA: 0s - loss: 1.8795 - accuracy: 0.3256
902/1563 [================>.............] - ETA: 0s - loss: 1.8751 - accuracy: 0.3271
940/1563 [=================>............] - ETA: 0s - loss: 1.8702 - accuracy: 0.3292
978/1563 [=================>............] - ETA: 0s - loss: 1.8654 - accuracy: 0.3311
1016/1563 [==================>...........] - ETA: 0s - loss: 1.8610 - accuracy: 0.3328
1053/1563 [===================>..........] - ETA: 0s - loss: 1.8568 - accuracy: 0.3347
1092/1563 [===================>..........] - ETA: 0s - loss: 1.8520 - accuracy: 0.3368
1130/1563 [====================>.........] - ETA: 0s - loss: 1.8493 - accuracy: 0.3381
1168/1563 [=====================>........] - ETA: 0s - loss: 1.8437 - accuracy: 0.3399
1206/1563 [======================>.......] - ETA: 0s - loss: 1.8400 - accuracy: 0.3410
1245/1563 [======================>.......] - ETA: 0s - loss: 1.8357 - accuracy: 0.3433
1283/1563 [=======================>......] - ETA: 0s - loss: 1.8308 - accuracy: 0.3448
1321/1563 [========================>.....] - ETA: 0s - loss: 1.8270 - accuracy: 0.3462
1359/1563 [=========================>....] - ETA: 0s - loss: 1.8252 - accuracy: 0.3468
1397/1563 [=========================>....] - ETA: 0s - loss: 1.8224 - accuracy: 0.3475
1435/1563 [==========================>...] - ETA: 0s - loss: 1.8212 - accuracy: 0.3483
1473/1563 [===========================>..] - ETA: 0s - loss: 1.8178 - accuracy: 0.3504
1512/1563 [============================>.] - ETA: 0s - loss: 1.8155 - accuracy: 0.3514
1550/1563 [============================>.] - ETA: 0s - loss: 1.8125 - accuracy: 0.3530
1563/1563 [==============================] - 5s 1ms/step - loss: 1.8115 - accuracy: 0.3531
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 1.6810 - accuracy: 0.3438
40/1563 [..............................] - ETA: 2s - loss: 1.6652 - accuracy: 0.4117
78/1563 [>.............................] - ETA: 1s - loss: 1.6621 - accuracy: 0.4247
116/1563 [=>............................] - ETA: 1s - loss: 1.6531 - accuracy: 0.4262
155/1563 [=>............................] - ETA: 1s - loss: 1.6527 - accuracy: 0.4290
193/1563 [==>...........................] - ETA: 1s - loss: 1.6592 - accuracy: 0.4244
231/1563 [===>..........................] - ETA: 1s - loss: 1.6611 - accuracy: 0.4229
270/1563 [====>.........................] - ETA: 1s - loss: 1.6632 - accuracy: 0.4198
308/1563 [====>.........................] - ETA: 1s - loss: 1.6620 - accuracy: 0.4182
346/1563 [=====>........................] - ETA: 1s - loss: 1.6642 - accuracy: 0.4161
384/1563 [======>.......................] - ETA: 1s - loss: 1.6596 - accuracy: 0.4159
422/1563 [=======>......................] - ETA: 1s - loss: 1.6582 - accuracy: 0.4142
459/1563 [=======>......................] - ETA: 1s - loss: 1.6553 - accuracy: 0.4153
497/1563 [========>.....................] - ETA: 1s - loss: 1.6504 - accuracy: 0.4167
535/1563 [=========>....................] - ETA: 1s - loss: 1.6505 - accuracy: 0.4165
573/1563 [=========>....................] - ETA: 1s - loss: 1.6460 - accuracy: 0.4189
611/1563 [==========>...................] - ETA: 1s - loss: 1.6464 - accuracy: 0.4194
649/1563 [===========>..................] - ETA: 1s - loss: 1.6454 - accuracy: 0.4196
687/1563 [============>.................] - ETA: 1s - loss: 1.6445 - accuracy: 0.4202
725/1563 [============>.................] - ETA: 1s - loss: 1.6428 - accuracy: 0.4203
763/1563 [=============>................] - ETA: 1s - loss: 1.6419 - accuracy: 0.4202
801/1563 [==============>...............] - ETA: 1s - loss: 1.6411 - accuracy: 0.4203
838/1563 [===============>..............] - ETA: 0s - loss: 1.6399 - accuracy: 0.4203
875/1563 [===============>..............] - ETA: 0s - loss: 1.6387 - accuracy: 0.4206
913/1563 [================>.............] - ETA: 0s - loss: 1.6374 - accuracy: 0.4211
951/1563 [=================>............] - ETA: 0s - loss: 1.6359 - accuracy: 0.4212
989/1563 [=================>............] - ETA: 0s - loss: 1.6348 - accuracy: 0.4215
1027/1563 [==================>...........] - ETA: 0s - loss: 1.6352 - accuracy: 0.4216
1064/1563 [===================>..........] - ETA: 0s - loss: 1.6342 - accuracy: 0.4218
1102/1563 [====================>.........] - ETA: 0s - loss: 1.6328 - accuracy: 0.4224
1140/1563 [====================>.........] - ETA: 0s - loss: 1.6331 - accuracy: 0.4227
1177/1563 [=====================>........] - ETA: 0s - loss: 1.6318 - accuracy: 0.4236
1215/1563 [======================>.......] - ETA: 0s - loss: 1.6308 - accuracy: 0.4242
1254/1563 [=======================>......] - ETA: 0s - loss: 1.6295 - accuracy: 0.4246
1291/1563 [=======================>......] - ETA: 0s - loss: 1.6288 - accuracy: 0.4245
1328/1563 [========================>.....] - ETA: 0s - loss: 1.6279 - accuracy: 0.4244
1365/1563 [=========================>....] - ETA: 0s - loss: 1.6279 - accuracy: 0.4247
1403/1563 [=========================>....] - ETA: 0s - loss: 1.6271 - accuracy: 0.4251
1442/1563 [==========================>...] - ETA: 0s - loss: 1.6271 - accuracy: 0.4252
1480/1563 [===========================>..] - ETA: 0s - loss: 1.6259 - accuracy: 0.4254
1517/1563 [============================>.] - ETA: 0s - loss: 1.6258 - accuracy: 0.4257
1555/1563 [============================>.] - ETA: 0s - loss: 1.6237 - accuracy: 0.4264
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6235 - accuracy: 0.4266
Epoch 3/10
1/1563 [..............................] - ETA: 2s - loss: 1.6282 - accuracy: 0.5312
39/1563 [..............................] - ETA: 2s - loss: 1.5285 - accuracy: 0.4583
77/1563 [>.............................] - ETA: 1s - loss: 1.5476 - accuracy: 0.4537
115/1563 [=>............................] - ETA: 1s - loss: 1.5522 - accuracy: 0.4516
153/1563 [=>............................] - ETA: 1s - loss: 1.5553 - accuracy: 0.4510
190/1563 [==>...........................] - ETA: 1s - loss: 1.5661 - accuracy: 0.4462
227/1563 [===>..........................] - ETA: 1s - loss: 1.5574 - accuracy: 0.4502
264/1563 [====>.........................] - ETA: 1s - loss: 1.5533 - accuracy: 0.4512
301/1563 [====>.........................] - ETA: 1s - loss: 1.5518 - accuracy: 0.4511
339/1563 [=====>........................] - ETA: 1s - loss: 1.5461 - accuracy: 0.4518
377/1563 [======>.......................] - ETA: 1s - loss: 1.5411 - accuracy: 0.4537
415/1563 [======>.......................] - ETA: 1s - loss: 1.5430 - accuracy: 0.4520
453/1563 [=======>......................] - ETA: 1s - loss: 1.5409 - accuracy: 0.4524
491/1563 [========>.....................] - ETA: 1s - loss: 1.5432 - accuracy: 0.4530
530/1563 [=========>....................] - ETA: 1s - loss: 1.5444 - accuracy: 0.4517
568/1563 [=========>....................] - ETA: 1s - loss: 1.5457 - accuracy: 0.4516
606/1563 [==========>...................] - ETA: 1s - loss: 1.5464 - accuracy: 0.4509
643/1563 [===========>..................] - ETA: 1s - loss: 1.5469 - accuracy: 0.4499
680/1563 [============>.................] - ETA: 1s - loss: 1.5489 - accuracy: 0.4492
718/1563 [============>.................] - ETA: 1s - loss: 1.5479 - accuracy: 0.4484
756/1563 [=============>................] - ETA: 1s - loss: 1.5476 - accuracy: 0.4488
793/1563 [==============>...............] - ETA: 1s - loss: 1.5490 - accuracy: 0.4484
831/1563 [==============>...............] - ETA: 0s - loss: 1.5504 - accuracy: 0.4479
869/1563 [===============>..............] - ETA: 0s - loss: 1.5488 - accuracy: 0.4483
906/1563 [================>.............] - ETA: 0s - loss: 1.5506 - accuracy: 0.4480
944/1563 [=================>............] - ETA: 0s - loss: 1.5498 - accuracy: 0.4490
982/1563 [=================>............] - ETA: 0s - loss: 1.5490 - accuracy: 0.4496
1020/1563 [==================>...........] - ETA: 0s - loss: 1.5494 - accuracy: 0.4493
1058/1563 [===================>..........] - ETA: 0s - loss: 1.5482 - accuracy: 0.4503
1096/1563 [====================>.........] - ETA: 0s - loss: 1.5490 - accuracy: 0.4502
1135/1563 [====================>.........] - ETA: 0s - loss: 1.5491 - accuracy: 0.4507
1173/1563 [=====================>........] - ETA: 0s - loss: 1.5488 - accuracy: 0.4512
1211/1563 [======================>.......] - ETA: 0s - loss: 1.5469 - accuracy: 0.4521
1249/1563 [======================>.......] - ETA: 0s - loss: 1.5469 - accuracy: 0.4523
1287/1563 [=======================>......] - ETA: 0s - loss: 1.5471 - accuracy: 0.4522
1325/1563 [========================>.....] - ETA: 0s - loss: 1.5470 - accuracy: 0.4523
1363/1563 [=========================>....] - ETA: 0s - loss: 1.5471 - accuracy: 0.4527
1401/1563 [=========================>....] - ETA: 0s - loss: 1.5466 - accuracy: 0.4530
1438/1563 [==========================>...] - ETA: 0s - loss: 1.5456 - accuracy: 0.4533
1476/1563 [===========================>..] - ETA: 0s - loss: 1.5460 - accuracy: 0.4532
1515/1563 [============================>.] - ETA: 0s - loss: 1.5451 - accuracy: 0.4535
1553/1563 [============================>.] - ETA: 0s - loss: 1.5434 - accuracy: 0.4538
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5429 - accuracy: 0.4541
Epoch 4/10
1/1563 [..............................] - ETA: 2s - loss: 1.9384 - accuracy: 0.2812
39/1563 [..............................] - ETA: 2s - loss: 1.5235 - accuracy: 0.4583
77/1563 [>.............................] - ETA: 1s - loss: 1.5128 - accuracy: 0.4712
114/1563 [=>............................] - ETA: 1s - loss: 1.4941 - accuracy: 0.4740
152/1563 [=>............................] - ETA: 1s - loss: 1.4951 - accuracy: 0.4764
190/1563 [==>...........................] - ETA: 1s - loss: 1.4954 - accuracy: 0.4755
228/1563 [===>..........................] - ETA: 1s - loss: 1.4904 - accuracy: 0.4752
265/1563 [====>.........................] - ETA: 1s - loss: 1.4936 - accuracy: 0.4746
302/1563 [====>.........................] - ETA: 1s - loss: 1.4919 - accuracy: 0.4760
337/1563 [=====>........................] - ETA: 1s - loss: 1.4935 - accuracy: 0.4754
374/1563 [======>.......................] - ETA: 1s - loss: 1.4941 - accuracy: 0.4780
411/1563 [======>.......................] - ETA: 1s - loss: 1.4965 - accuracy: 0.4789
449/1563 [=======>......................] - ETA: 1s - loss: 1.4951 - accuracy: 0.4788
486/1563 [========>.....................] - ETA: 1s - loss: 1.4964 - accuracy: 0.4785
523/1563 [=========>....................] - ETA: 1s - loss: 1.4931 - accuracy: 0.4781
561/1563 [=========>....................] - ETA: 1s - loss: 1.4887 - accuracy: 0.4786
599/1563 [==========>...................] - ETA: 1s - loss: 1.4870 - accuracy: 0.4791
637/1563 [===========>..................] - ETA: 1s - loss: 1.4866 - accuracy: 0.4791
675/1563 [===========>..................] - ETA: 1s - loss: 1.4856 - accuracy: 0.4791
713/1563 [============>.................] - ETA: 1s - loss: 1.4846 - accuracy: 0.4790
751/1563 [=============>................] - ETA: 1s - loss: 1.4842 - accuracy: 0.4790
788/1563 [==============>...............] - ETA: 1s - loss: 1.4845 - accuracy: 0.4787
826/1563 [==============>...............] - ETA: 0s - loss: 1.4825 - accuracy: 0.4794
863/1563 [===============>..............] - ETA: 0s - loss: 1.4828 - accuracy: 0.4783
901/1563 [================>.............] - ETA: 0s - loss: 1.4812 - accuracy: 0.4793
939/1563 [=================>............] - ETA: 0s - loss: 1.4819 - accuracy: 0.4783
978/1563 [=================>............] - ETA: 0s - loss: 1.4821 - accuracy: 0.4779
1016/1563 [==================>...........] - ETA: 0s - loss: 1.4808 - accuracy: 0.4789
1054/1563 [===================>..........] - ETA: 0s - loss: 1.4822 - accuracy: 0.4789
1092/1563 [===================>..........] - ETA: 0s - loss: 1.4817 - accuracy: 0.4792
1130/1563 [====================>.........] - ETA: 0s - loss: 1.4813 - accuracy: 0.4793
1168/1563 [=====================>........] - ETA: 0s - loss: 1.4812 - accuracy: 0.4792
1205/1563 [======================>.......] - ETA: 0s - loss: 1.4812 - accuracy: 0.4793
1243/1563 [======================>.......] - ETA: 0s - loss: 1.4816 - accuracy: 0.4795
1281/1563 [=======================>......] - ETA: 0s - loss: 1.4812 - accuracy: 0.4796
1319/1563 [========================>.....] - ETA: 0s - loss: 1.4815 - accuracy: 0.4797
1355/1563 [=========================>....] - ETA: 0s - loss: 1.4803 - accuracy: 0.4803
1392/1563 [=========================>....] - ETA: 0s - loss: 1.4807 - accuracy: 0.4799
1430/1563 [==========================>...] - ETA: 0s - loss: 1.4804 - accuracy: 0.4798
1468/1563 [===========================>..] - ETA: 0s - loss: 1.4812 - accuracy: 0.4794
1504/1563 [===========================>..] - ETA: 0s - loss: 1.4820 - accuracy: 0.4787
1540/1563 [============================>.] - ETA: 0s - loss: 1.4817 - accuracy: 0.4788
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4818 - accuracy: 0.4785
Epoch 5/10
1/1563 [..............................] - ETA: 2s - loss: 1.2487 - accuracy: 0.6562
38/1563 [..............................] - ETA: 2s - loss: 1.4092 - accuracy: 0.5107
74/1563 [>.............................] - ETA: 2s - loss: 1.4248 - accuracy: 0.5063
111/1563 [=>............................] - ETA: 2s - loss: 1.4281 - accuracy: 0.5051
148/1563 [=>............................] - ETA: 1s - loss: 1.4434 - accuracy: 0.4926
186/1563 [==>...........................] - ETA: 1s - loss: 1.4453 - accuracy: 0.4898
224/1563 [===>..........................] - ETA: 1s - loss: 1.4457 - accuracy: 0.4904
261/1563 [====>.........................] - ETA: 1s - loss: 1.4436 - accuracy: 0.4897
299/1563 [====>.........................] - ETA: 1s - loss: 1.4437 - accuracy: 0.4901
337/1563 [=====>........................] - ETA: 1s - loss: 1.4442 - accuracy: 0.4911
374/1563 [======>.......................] - ETA: 1s - loss: 1.4449 - accuracy: 0.4900
411/1563 [======>.......................] - ETA: 1s - loss: 1.4425 - accuracy: 0.4906
449/1563 [=======>......................] - ETA: 1s - loss: 1.4394 - accuracy: 0.4919
487/1563 [========>.....................] - ETA: 1s - loss: 1.4409 - accuracy: 0.4917
524/1563 [=========>....................] - ETA: 1s - loss: 1.4400 - accuracy: 0.4919
562/1563 [=========>....................] - ETA: 1s - loss: 1.4393 - accuracy: 0.4910
598/1563 [==========>...................] - ETA: 1s - loss: 1.4398 - accuracy: 0.4901
635/1563 [===========>..................] - ETA: 1s - loss: 1.4378 - accuracy: 0.4912
671/1563 [===========>..................] - ETA: 1s - loss: 1.4391 - accuracy: 0.4909
710/1563 [============>.................] - ETA: 1s - loss: 1.4398 - accuracy: 0.4911
746/1563 [=============>................] - ETA: 1s - loss: 1.4368 - accuracy: 0.4927
782/1563 [==============>...............] - ETA: 1s - loss: 1.4375 - accuracy: 0.4925
818/1563 [==============>...............] - ETA: 1s - loss: 1.4369 - accuracy: 0.4925
856/1563 [===============>..............] - ETA: 0s - loss: 1.4375 - accuracy: 0.4928
893/1563 [================>.............] - ETA: 0s - loss: 1.4368 - accuracy: 0.4930
926/1563 [================>.............] - ETA: 0s - loss: 1.4383 - accuracy: 0.4929
962/1563 [=================>............] - ETA: 0s - loss: 1.4380 - accuracy: 0.4931
1000/1563 [==================>...........] - ETA: 0s - loss: 1.4380 - accuracy: 0.4929
1038/1563 [==================>...........] - ETA: 0s - loss: 1.4360 - accuracy: 0.4942
1073/1563 [===================>..........] - ETA: 0s - loss: 1.4357 - accuracy: 0.4945
1111/1563 [====================>.........] - ETA: 0s - loss: 1.4356 - accuracy: 0.4947
1149/1563 [=====================>........] - ETA: 0s - loss: 1.4357 - accuracy: 0.4941
1187/1563 [=====================>........] - ETA: 0s - loss: 1.4356 - accuracy: 0.4942
1225/1563 [======================>.......] - ETA: 0s - loss: 1.4356 - accuracy: 0.4940
1263/1563 [=======================>......] - ETA: 0s - loss: 1.4360 - accuracy: 0.4940
1301/1563 [=======================>......] - ETA: 0s - loss: 1.4354 - accuracy: 0.4939
1338/1563 [========================>.....] - ETA: 0s - loss: 1.4334 - accuracy: 0.4944
1376/1563 [=========================>....] - ETA: 0s - loss: 1.4327 - accuracy: 0.4949
1414/1563 [==========================>...] - ETA: 0s - loss: 1.4336 - accuracy: 0.4946
1452/1563 [==========================>...] - ETA: 0s - loss: 1.4336 - accuracy: 0.4944
1490/1563 [===========================>..] - ETA: 0s - loss: 1.4340 - accuracy: 0.4940
1528/1563 [============================>.] - ETA: 0s - loss: 1.4349 - accuracy: 0.4934
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4343 - accuracy: 0.4935
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.4592 - accuracy: 0.3750
38/1563 [..............................] - ETA: 2s - loss: 1.4246 - accuracy: 0.4901
75/1563 [>.............................] - ETA: 2s - loss: 1.3912 - accuracy: 0.5117
112/1563 [=>............................] - ETA: 1s - loss: 1.4010 - accuracy: 0.5070
150/1563 [=>............................] - ETA: 1s - loss: 1.4106 - accuracy: 0.5033
188/1563 [==>...........................] - ETA: 1s - loss: 1.4104 - accuracy: 0.5035
226/1563 [===>..........................] - ETA: 1s - loss: 1.4077 - accuracy: 0.5036
264/1563 [====>.........................] - ETA: 1s - loss: 1.4009 - accuracy: 0.5075
302/1563 [====>.........................] - ETA: 1s - loss: 1.3980 - accuracy: 0.5073
340/1563 [=====>........................] - ETA: 1s - loss: 1.4019 - accuracy: 0.5080
377/1563 [======>.......................] - ETA: 1s - loss: 1.4046 - accuracy: 0.5070
414/1563 [======>.......................] - ETA: 1s - loss: 1.4003 - accuracy: 0.5087
453/1563 [=======>......................] - ETA: 1s - loss: 1.4034 - accuracy: 0.5070
490/1563 [========>.....................] - ETA: 1s - loss: 1.4021 - accuracy: 0.5091
528/1563 [=========>....................] - ETA: 1s - loss: 1.3996 - accuracy: 0.5106
566/1563 [=========>....................] - ETA: 1s - loss: 1.4021 - accuracy: 0.5100
604/1563 [==========>...................] - ETA: 1s - loss: 1.4005 - accuracy: 0.5092
641/1563 [===========>..................] - ETA: 1s - loss: 1.3994 - accuracy: 0.5086
678/1563 [============>.................] - ETA: 1s - loss: 1.3985 - accuracy: 0.5087
716/1563 [============>.................] - ETA: 1s - loss: 1.3970 - accuracy: 0.5100
753/1563 [=============>................] - ETA: 1s - loss: 1.3986 - accuracy: 0.5085
790/1563 [==============>...............] - ETA: 1s - loss: 1.3996 - accuracy: 0.5071
828/1563 [==============>...............] - ETA: 0s - loss: 1.4005 - accuracy: 0.5063
866/1563 [===============>..............] - ETA: 0s - loss: 1.3990 - accuracy: 0.5065
903/1563 [================>.............] - ETA: 0s - loss: 1.4015 - accuracy: 0.5047
941/1563 [=================>............] - ETA: 0s - loss: 1.3998 - accuracy: 0.5057
979/1563 [=================>............] - ETA: 0s - loss: 1.3986 - accuracy: 0.5060
1016/1563 [==================>...........] - ETA: 0s - loss: 1.3974 - accuracy: 0.5063
1053/1563 [===================>..........] - ETA: 0s - loss: 1.3969 - accuracy: 0.5066
1091/1563 [===================>..........] - ETA: 0s - loss: 1.3963 - accuracy: 0.5067
1129/1563 [====================>.........] - ETA: 0s - loss: 1.3962 - accuracy: 0.5069
1166/1563 [=====================>........] - ETA: 0s - loss: 1.3955 - accuracy: 0.5079
1204/1563 [======================>.......] - ETA: 0s - loss: 1.3941 - accuracy: 0.5084
1242/1563 [======================>.......] - ETA: 0s - loss: 1.3954 - accuracy: 0.5071
1280/1563 [=======================>......] - ETA: 0s - loss: 1.3943 - accuracy: 0.5074
1318/1563 [========================>.....] - ETA: 0s - loss: 1.3950 - accuracy: 0.5074
1356/1563 [=========================>....] - ETA: 0s - loss: 1.3959 - accuracy: 0.5071
1393/1563 [=========================>....] - ETA: 0s - loss: 1.3952 - accuracy: 0.5076
1431/1563 [==========================>...] - ETA: 0s - loss: 1.3950 - accuracy: 0.5076
1470/1563 [===========================>..] - ETA: 0s - loss: 1.3950 - accuracy: 0.5078
1507/1563 [===========================>..] - ETA: 0s - loss: 1.3942 - accuracy: 0.5082
1545/1563 [============================>.] - ETA: 0s - loss: 1.3930 - accuracy: 0.5086
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3933 - accuracy: 0.5083
Epoch 7/10
1/1563 [..............................] - ETA: 2s - loss: 1.2119 - accuracy: 0.6250
39/1563 [..............................] - ETA: 2s - loss: 1.3829 - accuracy: 0.5032
77/1563 [>.............................] - ETA: 1s - loss: 1.3634 - accuracy: 0.5170
113/1563 [=>............................] - ETA: 1s - loss: 1.3645 - accuracy: 0.5166
150/1563 [=>............................] - ETA: 1s - loss: 1.3567 - accuracy: 0.5167
188/1563 [==>...........................] - ETA: 1s - loss: 1.3573 - accuracy: 0.5176
225/1563 [===>..........................] - ETA: 1s - loss: 1.3616 - accuracy: 0.5171
263/1563 [====>.........................] - ETA: 1s - loss: 1.3581 - accuracy: 0.5191
300/1563 [====>.........................] - ETA: 1s - loss: 1.3618 - accuracy: 0.5189
337/1563 [=====>........................] - ETA: 1s - loss: 1.3644 - accuracy: 0.5182
375/1563 [======>.......................] - ETA: 1s - loss: 1.3657 - accuracy: 0.5168
412/1563 [======>.......................] - ETA: 1s - loss: 1.3675 - accuracy: 0.5172
450/1563 [=======>......................] - ETA: 1s - loss: 1.3646 - accuracy: 0.5192
488/1563 [========>.....................] - ETA: 1s - loss: 1.3635 - accuracy: 0.5210
526/1563 [=========>....................] - ETA: 1s - loss: 1.3631 - accuracy: 0.5212
563/1563 [=========>....................] - ETA: 1s - loss: 1.3634 - accuracy: 0.5203
601/1563 [==========>...................] - ETA: 1s - loss: 1.3635 - accuracy: 0.5209
639/1563 [===========>..................] - ETA: 1s - loss: 1.3620 - accuracy: 0.5216
676/1563 [===========>..................] - ETA: 1s - loss: 1.3647 - accuracy: 0.5207
713/1563 [============>.................] - ETA: 1s - loss: 1.3648 - accuracy: 0.5207
751/1563 [=============>................] - ETA: 1s - loss: 1.3637 - accuracy: 0.5210
788/1563 [==============>...............] - ETA: 1s - loss: 1.3656 - accuracy: 0.5198
827/1563 [==============>...............] - ETA: 0s - loss: 1.3642 - accuracy: 0.5195
865/1563 [===============>..............] - ETA: 0s - loss: 1.3653 - accuracy: 0.5183
903/1563 [================>.............] - ETA: 0s - loss: 1.3651 - accuracy: 0.5179
941/1563 [=================>............] - ETA: 0s - loss: 1.3622 - accuracy: 0.5189
979/1563 [=================>............] - ETA: 0s - loss: 1.3613 - accuracy: 0.5191
1017/1563 [==================>...........] - ETA: 0s - loss: 1.3611 - accuracy: 0.5194
1054/1563 [===================>..........] - ETA: 0s - loss: 1.3615 - accuracy: 0.5199
1092/1563 [===================>..........] - ETA: 0s - loss: 1.3621 - accuracy: 0.5194
1130/1563 [====================>.........] - ETA: 0s - loss: 1.3607 - accuracy: 0.5201
1168/1563 [=====================>........] - ETA: 0s - loss: 1.3586 - accuracy: 0.5210
1206/1563 [======================>.......] - ETA: 0s - loss: 1.3572 - accuracy: 0.5214
1243/1563 [======================>.......] - ETA: 0s - loss: 1.3577 - accuracy: 0.5212
1281/1563 [=======================>......] - ETA: 0s - loss: 1.3564 - accuracy: 0.5214
1319/1563 [========================>.....] - ETA: 0s - loss: 1.3560 - accuracy: 0.5211
1357/1563 [=========================>....] - ETA: 0s - loss: 1.3559 - accuracy: 0.5214
1394/1563 [=========================>....] - ETA: 0s - loss: 1.3551 - accuracy: 0.5216
1432/1563 [==========================>...] - ETA: 0s - loss: 1.3546 - accuracy: 0.5216
1470/1563 [===========================>..] - ETA: 0s - loss: 1.3549 - accuracy: 0.5216
1508/1563 [===========================>..] - ETA: 0s - loss: 1.3536 - accuracy: 0.5223
1545/1563 [============================>.] - ETA: 0s - loss: 1.3528 - accuracy: 0.5225
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3535 - accuracy: 0.5224
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.1541 - accuracy: 0.6250
39/1563 [..............................] - ETA: 2s - loss: 1.3391 - accuracy: 0.5232
76/1563 [>.............................] - ETA: 2s - loss: 1.3204 - accuracy: 0.5321
114/1563 [=>............................] - ETA: 1s - loss: 1.2972 - accuracy: 0.5471
152/1563 [=>............................] - ETA: 1s - loss: 1.2994 - accuracy: 0.5481
189/1563 [==>...........................] - ETA: 1s - loss: 1.3083 - accuracy: 0.5463
227/1563 [===>..........................] - ETA: 1s - loss: 1.3094 - accuracy: 0.5441
265/1563 [====>.........................] - ETA: 1s - loss: 1.3107 - accuracy: 0.5430
302/1563 [====>.........................] - ETA: 1s - loss: 1.3124 - accuracy: 0.5424
340/1563 [=====>........................] - ETA: 1s - loss: 1.3157 - accuracy: 0.5417
378/1563 [======>.......................] - ETA: 1s - loss: 1.3175 - accuracy: 0.5408
416/1563 [======>.......................] - ETA: 1s - loss: 1.3180 - accuracy: 0.5408
454/1563 [=======>......................] - ETA: 1s - loss: 1.3173 - accuracy: 0.5412
491/1563 [========>.....................] - ETA: 1s - loss: 1.3176 - accuracy: 0.5407
529/1563 [=========>....................] - ETA: 1s - loss: 1.3192 - accuracy: 0.5396
567/1563 [=========>....................] - ETA: 1s - loss: 1.3195 - accuracy: 0.5396
605/1563 [==========>...................] - ETA: 1s - loss: 1.3210 - accuracy: 0.5389
641/1563 [===========>..................] - ETA: 1s - loss: 1.3216 - accuracy: 0.5385
679/1563 [============>.................] - ETA: 1s - loss: 1.3254 - accuracy: 0.5362
718/1563 [============>.................] - ETA: 1s - loss: 1.3281 - accuracy: 0.5347
756/1563 [=============>................] - ETA: 1s - loss: 1.3269 - accuracy: 0.5351
793/1563 [==============>...............] - ETA: 1s - loss: 1.3237 - accuracy: 0.5364
829/1563 [==============>...............] - ETA: 0s - loss: 1.3244 - accuracy: 0.5358
866/1563 [===============>..............] - ETA: 0s - loss: 1.3243 - accuracy: 0.5355
904/1563 [================>.............] - ETA: 0s - loss: 1.3267 - accuracy: 0.5346
942/1563 [=================>............] - ETA: 0s - loss: 1.3250 - accuracy: 0.5353
979/1563 [=================>............] - ETA: 0s - loss: 1.3228 - accuracy: 0.5361
1017/1563 [==================>...........] - ETA: 0s - loss: 1.3226 - accuracy: 0.5368
1054/1563 [===================>..........] - ETA: 0s - loss: 1.3239 - accuracy: 0.5363
1091/1563 [===================>..........] - ETA: 0s - loss: 1.3239 - accuracy: 0.5365
1129/1563 [====================>.........] - ETA: 0s - loss: 1.3229 - accuracy: 0.5364
1167/1563 [=====================>........] - ETA: 0s - loss: 1.3215 - accuracy: 0.5365
1205/1563 [======================>.......] - ETA: 0s - loss: 1.3204 - accuracy: 0.5369
1243/1563 [======================>.......] - ETA: 0s - loss: 1.3203 - accuracy: 0.5372
1280/1563 [=======================>......] - ETA: 0s - loss: 1.3206 - accuracy: 0.5371
1318/1563 [========================>.....] - ETA: 0s - loss: 1.3200 - accuracy: 0.5372
1355/1563 [=========================>....] - ETA: 0s - loss: 1.3182 - accuracy: 0.5379
1393/1563 [=========================>....] - ETA: 0s - loss: 1.3172 - accuracy: 0.5381
1430/1563 [==========================>...] - ETA: 0s - loss: 1.3165 - accuracy: 0.5383
1467/1563 [===========================>..] - ETA: 0s - loss: 1.3161 - accuracy: 0.5380
1504/1563 [===========================>..] - ETA: 0s - loss: 1.3165 - accuracy: 0.5382
1541/1563 [============================>.] - ETA: 0s - loss: 1.3178 - accuracy: 0.5377
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3173 - accuracy: 0.5377
Epoch 9/10
1/1563 [..............................] - ETA: 2s - loss: 1.3291 - accuracy: 0.5312
38/1563 [..............................] - ETA: 2s - loss: 1.2964 - accuracy: 0.5403
75/1563 [>.............................] - ETA: 2s - loss: 1.2915 - accuracy: 0.5321
113/1563 [=>............................] - ETA: 1s - loss: 1.2851 - accuracy: 0.5387
151/1563 [=>............................] - ETA: 1s - loss: 1.2967 - accuracy: 0.5373
189/1563 [==>...........................] - ETA: 1s - loss: 1.2957 - accuracy: 0.5380
227/1563 [===>..........................] - ETA: 1s - loss: 1.2886 - accuracy: 0.5425
264/1563 [====>.........................] - ETA: 1s - loss: 1.2850 - accuracy: 0.5434
302/1563 [====>.........................] - ETA: 1s - loss: 1.2907 - accuracy: 0.5423
339/1563 [=====>........................] - ETA: 1s - loss: 1.2923 - accuracy: 0.5415
377/1563 [======>.......................] - ETA: 1s - loss: 1.2921 - accuracy: 0.5431
415/1563 [======>.......................] - ETA: 1s - loss: 1.2950 - accuracy: 0.5428
452/1563 [=======>......................] - ETA: 1s - loss: 1.2921 - accuracy: 0.5434
489/1563 [========>.....................] - ETA: 1s - loss: 1.2862 - accuracy: 0.5477
526/1563 [=========>....................] - ETA: 1s - loss: 1.2854 - accuracy: 0.5477
564/1563 [=========>....................] - ETA: 1s - loss: 1.2871 - accuracy: 0.5469
602/1563 [==========>...................] - ETA: 1s - loss: 1.2858 - accuracy: 0.5487
640/1563 [===========>..................] - ETA: 1s - loss: 1.2822 - accuracy: 0.5502
678/1563 [============>.................] - ETA: 1s - loss: 1.2848 - accuracy: 0.5484
716/1563 [============>.................] - ETA: 1s - loss: 1.2854 - accuracy: 0.5484
754/1563 [=============>................] - ETA: 1s - loss: 1.2877 - accuracy: 0.5478
792/1563 [==============>...............] - ETA: 1s - loss: 1.2857 - accuracy: 0.5479
830/1563 [==============>...............] - ETA: 0s - loss: 1.2864 - accuracy: 0.5484
867/1563 [===============>..............] - ETA: 0s - loss: 1.2885 - accuracy: 0.5480
904/1563 [================>.............] - ETA: 0s - loss: 1.2873 - accuracy: 0.5489
942/1563 [=================>............] - ETA: 0s - loss: 1.2867 - accuracy: 0.5489
980/1563 [=================>............] - ETA: 0s - loss: 1.2877 - accuracy: 0.5480
1018/1563 [==================>...........] - ETA: 0s - loss: 1.2880 - accuracy: 0.5474
1055/1563 [===================>..........] - ETA: 0s - loss: 1.2892 - accuracy: 0.5476
1093/1563 [===================>..........] - ETA: 0s - loss: 1.2886 - accuracy: 0.5475
1130/1563 [====================>.........] - ETA: 0s - loss: 1.2882 - accuracy: 0.5478
1168/1563 [=====================>........] - ETA: 0s - loss: 1.2857 - accuracy: 0.5488
1205/1563 [======================>.......] - ETA: 0s - loss: 1.2835 - accuracy: 0.5494
1242/1563 [======================>.......] - ETA: 0s - loss: 1.2836 - accuracy: 0.5492
1280/1563 [=======================>......] - ETA: 0s - loss: 1.2858 - accuracy: 0.5486
1319/1563 [========================>.....] - ETA: 0s - loss: 1.2859 - accuracy: 0.5480
1357/1563 [=========================>....] - ETA: 0s - loss: 1.2852 - accuracy: 0.5487
1394/1563 [=========================>....] - ETA: 0s - loss: 1.2863 - accuracy: 0.5482
1432/1563 [==========================>...] - ETA: 0s - loss: 1.2857 - accuracy: 0.5485
1470/1563 [===========================>..] - ETA: 0s - loss: 1.2867 - accuracy: 0.5478
1508/1563 [===========================>..] - ETA: 0s - loss: 1.2863 - accuracy: 0.5485
1546/1563 [============================>.] - ETA: 0s - loss: 1.2857 - accuracy: 0.5487
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2858 - accuracy: 0.5488
Epoch 10/10
1/1563 [..............................] - ETA: 2s - loss: 1.7119 - accuracy: 0.4688
39/1563 [..............................] - ETA: 2s - loss: 1.2738 - accuracy: 0.5401
76/1563 [>.............................] - ETA: 1s - loss: 1.2675 - accuracy: 0.5411
114/1563 [=>............................] - ETA: 1s - loss: 1.2584 - accuracy: 0.5507
151/1563 [=>............................] - ETA: 1s - loss: 1.2511 - accuracy: 0.5582
188/1563 [==>...........................] - ETA: 1s - loss: 1.2573 - accuracy: 0.5567
225/1563 [===>..........................] - ETA: 1s - loss: 1.2508 - accuracy: 0.5596
262/1563 [====>.........................] - ETA: 1s - loss: 1.2573 - accuracy: 0.5611
300/1563 [====>.........................] - ETA: 1s - loss: 1.2606 - accuracy: 0.5603
337/1563 [=====>........................] - ETA: 1s - loss: 1.2600 - accuracy: 0.5591
375/1563 [======>.......................] - ETA: 1s - loss: 1.2548 - accuracy: 0.5592
412/1563 [======>.......................] - ETA: 1s - loss: 1.2525 - accuracy: 0.5604
449/1563 [=======>......................] - ETA: 1s - loss: 1.2501 - accuracy: 0.5615
487/1563 [========>.....................] - ETA: 1s - loss: 1.2512 - accuracy: 0.5597
525/1563 [=========>....................] - ETA: 1s - loss: 1.2500 - accuracy: 0.5595
563/1563 [=========>....................] - ETA: 1s - loss: 1.2506 - accuracy: 0.5595
601/1563 [==========>...................] - ETA: 1s - loss: 1.2501 - accuracy: 0.5604
639/1563 [===========>..................] - ETA: 1s - loss: 1.2496 - accuracy: 0.5596
677/1563 [===========>..................] - ETA: 1s - loss: 1.2488 - accuracy: 0.5596
715/1563 [============>.................] - ETA: 1s - loss: 1.2512 - accuracy: 0.5597
753/1563 [=============>................] - ETA: 1s - loss: 1.2532 - accuracy: 0.5590
791/1563 [==============>...............] - ETA: 1s - loss: 1.2531 - accuracy: 0.5595
829/1563 [==============>...............] - ETA: 0s - loss: 1.2543 - accuracy: 0.5587
867/1563 [===============>..............] - ETA: 0s - loss: 1.2567 - accuracy: 0.5582
905/1563 [================>.............] - ETA: 0s - loss: 1.2564 - accuracy: 0.5580
942/1563 [=================>............] - ETA: 0s - loss: 1.2569 - accuracy: 0.5581
979/1563 [=================>............] - ETA: 0s - loss: 1.2588 - accuracy: 0.5577
1016/1563 [==================>...........] - ETA: 0s - loss: 1.2595 - accuracy: 0.5577
1053/1563 [===================>..........] - ETA: 0s - loss: 1.2588 - accuracy: 0.5581
1090/1563 [===================>..........] - ETA: 0s - loss: 1.2584 - accuracy: 0.5587
1128/1563 [====================>.........] - ETA: 0s - loss: 1.2566 - accuracy: 0.5594
1166/1563 [=====================>........] - ETA: 0s - loss: 1.2574 - accuracy: 0.5592
1204/1563 [======================>.......] - ETA: 0s - loss: 1.2561 - accuracy: 0.5596
1241/1563 [======================>.......] - ETA: 0s - loss: 1.2559 - accuracy: 0.5595
1279/1563 [=======================>......] - ETA: 0s - loss: 1.2553 - accuracy: 0.5594
1316/1563 [========================>.....] - ETA: 0s - loss: 1.2539 - accuracy: 0.5600
1353/1563 [========================>.....] - ETA: 0s - loss: 1.2557 - accuracy: 0.5591
1391/1563 [=========================>....] - ETA: 0s - loss: 1.2557 - accuracy: 0.5587
1428/1563 [==========================>...] - ETA: 0s - loss: 1.2568 - accuracy: 0.5586
1465/1563 [===========================>..] - ETA: 0s - loss: 1.2563 - accuracy: 0.5590
1502/1563 [===========================>..] - ETA: 0s - loss: 1.2564 - accuracy: 0.5588
1538/1563 [============================>.] - ETA: 0s - loss: 1.2569 - accuracy: 0.5588
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2566 - accuracy: 0.5589
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 27.183104515075684
hydro05
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins2632562304256685857.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
