Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 90372 queued and waiting for resources
srun: job 90372 has been allocated resources
2024-10-21 07:43:08.462742: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-21 07:43:32.137242: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-21 07:44:09.300840: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-10-21 07:44:09.301384: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-10-21 07:44:09.301392: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-10-21 07:45:09.492348: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-21 07:45:11.838264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-10-21 07:45:20.747372: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 1:53:58 - loss: 2.2386 - accuracy: 0.2500
38/1563 [..............................] - ETA: 2s - loss: 2.2541 - accuracy: 0.1637
76/1563 [>.............................] - ETA: 2s - loss: 2.1697 - accuracy: 0.2015
114/1563 [=>............................] - ETA: 1s - loss: 2.1130 - accuracy: 0.2248
153/1563 [=>............................] - ETA: 1s - loss: 2.0809 - accuracy: 0.2394
191/1563 [==>...........................] - ETA: 1s - loss: 2.0506 - accuracy: 0.2536
230/1563 [===>..........................] - ETA: 1s - loss: 2.0313 - accuracy: 0.2621
269/1563 [====>.........................] - ETA: 1s - loss: 2.0068 - accuracy: 0.2743
308/1563 [====>.........................] - ETA: 1s - loss: 1.9906 - accuracy: 0.2815
346/1563 [=====>........................] - ETA: 1s - loss: 1.9749 - accuracy: 0.2893
384/1563 [======>.......................] - ETA: 1s - loss: 1.9650 - accuracy: 0.2927
423/1563 [=======>......................] - ETA: 1s - loss: 1.9599 - accuracy: 0.2937
461/1563 [=======>......................] - ETA: 1s - loss: 1.9510 - accuracy: 0.2970
500/1563 [========>.....................] - ETA: 1s - loss: 1.9392 - accuracy: 0.3011
539/1563 [=========>....................] - ETA: 1s - loss: 1.9302 - accuracy: 0.3045
578/1563 [==========>...................] - ETA: 1s - loss: 1.9237 - accuracy: 0.3063
617/1563 [==========>...................] - ETA: 1s - loss: 1.9184 - accuracy: 0.3082
656/1563 [===========>..................] - ETA: 1s - loss: 1.9095 - accuracy: 0.3123
695/1563 [============>.................] - ETA: 1s - loss: 1.9028 - accuracy: 0.3153
734/1563 [=============>................] - ETA: 1s - loss: 1.8957 - accuracy: 0.3192
773/1563 [=============>................] - ETA: 1s - loss: 1.8895 - accuracy: 0.3219
812/1563 [==============>...............] - ETA: 0s - loss: 1.8837 - accuracy: 0.3252
851/1563 [===============>..............] - ETA: 0s - loss: 1.8782 - accuracy: 0.3268
890/1563 [================>.............] - ETA: 0s - loss: 1.8732 - accuracy: 0.3291
929/1563 [================>.............] - ETA: 0s - loss: 1.8705 - accuracy: 0.3311
968/1563 [=================>............] - ETA: 0s - loss: 1.8647 - accuracy: 0.3336
1007/1563 [==================>...........] - ETA: 0s - loss: 1.8600 - accuracy: 0.3352
1045/1563 [===================>..........] - ETA: 0s - loss: 1.8550 - accuracy: 0.3372
1083/1563 [===================>..........] - ETA: 0s - loss: 1.8510 - accuracy: 0.3399
1121/1563 [====================>.........] - ETA: 0s - loss: 1.8483 - accuracy: 0.3407
1159/1563 [=====================>........] - ETA: 0s - loss: 1.8441 - accuracy: 0.3425
1197/1563 [=====================>........] - ETA: 0s - loss: 1.8402 - accuracy: 0.3439
1235/1563 [======================>.......] - ETA: 0s - loss: 1.8364 - accuracy: 0.3460
1273/1563 [=======================>......] - ETA: 0s - loss: 1.8324 - accuracy: 0.3475
1311/1563 [========================>.....] - ETA: 0s - loss: 1.8291 - accuracy: 0.3483
1349/1563 [========================>.....] - ETA: 0s - loss: 1.8259 - accuracy: 0.3492
1387/1563 [=========================>....] - ETA: 0s - loss: 1.8232 - accuracy: 0.3501
1425/1563 [==========================>...] - ETA: 0s - loss: 1.8198 - accuracy: 0.3513
1463/1563 [===========================>..] - ETA: 0s - loss: 1.8160 - accuracy: 0.3524
1501/1563 [===========================>..] - ETA: 0s - loss: 1.8124 - accuracy: 0.3540
1539/1563 [============================>.] - ETA: 0s - loss: 1.8090 - accuracy: 0.3555
1563/1563 [==============================] - 6s 1ms/step - loss: 1.8075 - accuracy: 0.3562
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 2.1522 - accuracy: 0.1562
39/1563 [..............................] - ETA: 2s - loss: 1.6532 - accuracy: 0.4207
77/1563 [>.............................] - ETA: 1s - loss: 1.6523 - accuracy: 0.4265
115/1563 [=>............................] - ETA: 1s - loss: 1.6594 - accuracy: 0.4168
153/1563 [=>............................] - ETA: 1s - loss: 1.6490 - accuracy: 0.4173
192/1563 [==>...........................] - ETA: 1s - loss: 1.6467 - accuracy: 0.4165
230/1563 [===>..........................] - ETA: 1s - loss: 1.6459 - accuracy: 0.4140
268/1563 [====>.........................] - ETA: 1s - loss: 1.6436 - accuracy: 0.4159
306/1563 [====>.........................] - ETA: 1s - loss: 1.6441 - accuracy: 0.4161
344/1563 [=====>........................] - ETA: 1s - loss: 1.6436 - accuracy: 0.4156
382/1563 [======>.......................] - ETA: 1s - loss: 1.6431 - accuracy: 0.4136
420/1563 [=======>......................] - ETA: 1s - loss: 1.6436 - accuracy: 0.4140
459/1563 [=======>......................] - ETA: 1s - loss: 1.6402 - accuracy: 0.4156
498/1563 [========>.....................] - ETA: 1s - loss: 1.6370 - accuracy: 0.4170
537/1563 [=========>....................] - ETA: 1s - loss: 1.6384 - accuracy: 0.4160
576/1563 [==========>...................] - ETA: 1s - loss: 1.6389 - accuracy: 0.4159
615/1563 [==========>...................] - ETA: 1s - loss: 1.6392 - accuracy: 0.4151
653/1563 [===========>..................] - ETA: 1s - loss: 1.6388 - accuracy: 0.4160
691/1563 [============>.................] - ETA: 1s - loss: 1.6394 - accuracy: 0.4161
719/1563 [============>.................] - ETA: 1s - loss: 1.6387 - accuracy: 0.4165
757/1563 [=============>................] - ETA: 1s - loss: 1.6373 - accuracy: 0.4174
795/1563 [==============>...............] - ETA: 1s - loss: 1.6355 - accuracy: 0.4186
833/1563 [==============>...............] - ETA: 0s - loss: 1.6361 - accuracy: 0.4181
871/1563 [===============>..............] - ETA: 0s - loss: 1.6349 - accuracy: 0.4190
909/1563 [================>.............] - ETA: 0s - loss: 1.6339 - accuracy: 0.4194
947/1563 [=================>............] - ETA: 0s - loss: 1.6323 - accuracy: 0.4198
985/1563 [=================>............] - ETA: 0s - loss: 1.6294 - accuracy: 0.4214
1023/1563 [==================>...........] - ETA: 0s - loss: 1.6284 - accuracy: 0.4222
1061/1563 [===================>..........] - ETA: 0s - loss: 1.6276 - accuracy: 0.4224
1099/1563 [====================>.........] - ETA: 0s - loss: 1.6278 - accuracy: 0.4223
1137/1563 [====================>.........] - ETA: 0s - loss: 1.6280 - accuracy: 0.4223
1175/1563 [=====================>........] - ETA: 0s - loss: 1.6265 - accuracy: 0.4227
1213/1563 [======================>.......] - ETA: 0s - loss: 1.6252 - accuracy: 0.4235
1252/1563 [=======================>......] - ETA: 0s - loss: 1.6249 - accuracy: 0.4238
1290/1563 [=======================>......] - ETA: 0s - loss: 1.6236 - accuracy: 0.4244
1328/1563 [========================>.....] - ETA: 0s - loss: 1.6235 - accuracy: 0.4241
1366/1563 [=========================>....] - ETA: 0s - loss: 1.6217 - accuracy: 0.4249
1404/1563 [=========================>....] - ETA: 0s - loss: 1.6212 - accuracy: 0.4252
1443/1563 [==========================>...] - ETA: 0s - loss: 1.6204 - accuracy: 0.4258
1482/1563 [===========================>..] - ETA: 0s - loss: 1.6200 - accuracy: 0.4262
1521/1563 [============================>.] - ETA: 0s - loss: 1.6195 - accuracy: 0.4265
1559/1563 [============================>.] - ETA: 0s - loss: 1.6198 - accuracy: 0.4267
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6197 - accuracy: 0.4269
Epoch 3/10
1/1563 [..............................] - ETA: 2s - loss: 1.5227 - accuracy: 0.3438
40/1563 [..............................] - ETA: 1s - loss: 1.5790 - accuracy: 0.4555
78/1563 [>.............................] - ETA: 1s - loss: 1.5815 - accuracy: 0.4555
117/1563 [=>............................] - ETA: 1s - loss: 1.5726 - accuracy: 0.4498
155/1563 [=>............................] - ETA: 1s - loss: 1.5784 - accuracy: 0.4437
194/1563 [==>...........................] - ETA: 1s - loss: 1.5730 - accuracy: 0.4439
233/1563 [===>..........................] - ETA: 1s - loss: 1.5662 - accuracy: 0.4497
272/1563 [====>.........................] - ETA: 1s - loss: 1.5623 - accuracy: 0.4497
310/1563 [====>.........................] - ETA: 1s - loss: 1.5648 - accuracy: 0.4469
349/1563 [=====>........................] - ETA: 1s - loss: 1.5626 - accuracy: 0.4477
388/1563 [======>.......................] - ETA: 1s - loss: 1.5653 - accuracy: 0.4473
427/1563 [=======>......................] - ETA: 1s - loss: 1.5616 - accuracy: 0.4483
466/1563 [=======>......................] - ETA: 1s - loss: 1.5602 - accuracy: 0.4488
505/1563 [========>.....................] - ETA: 1s - loss: 1.5566 - accuracy: 0.4507
544/1563 [=========>....................] - ETA: 1s - loss: 1.5533 - accuracy: 0.4509
583/1563 [==========>...................] - ETA: 1s - loss: 1.5576 - accuracy: 0.4483
621/1563 [==========>...................] - ETA: 1s - loss: 1.5563 - accuracy: 0.4484
659/1563 [===========>..................] - ETA: 1s - loss: 1.5541 - accuracy: 0.4489
697/1563 [============>.................] - ETA: 1s - loss: 1.5523 - accuracy: 0.4503
735/1563 [=============>................] - ETA: 1s - loss: 1.5508 - accuracy: 0.4510
773/1563 [=============>................] - ETA: 1s - loss: 1.5468 - accuracy: 0.4521
811/1563 [==============>...............] - ETA: 0s - loss: 1.5463 - accuracy: 0.4531
849/1563 [===============>..............] - ETA: 0s - loss: 1.5449 - accuracy: 0.4540
887/1563 [================>.............] - ETA: 0s - loss: 1.5437 - accuracy: 0.4542
925/1563 [================>.............] - ETA: 0s - loss: 1.5440 - accuracy: 0.4540
963/1563 [=================>............] - ETA: 0s - loss: 1.5435 - accuracy: 0.4547
1001/1563 [==================>...........] - ETA: 0s - loss: 1.5427 - accuracy: 0.4552
1039/1563 [==================>...........] - ETA: 0s - loss: 1.5448 - accuracy: 0.4547
1076/1563 [===================>..........] - ETA: 0s - loss: 1.5456 - accuracy: 0.4546
1114/1563 [====================>.........] - ETA: 0s - loss: 1.5450 - accuracy: 0.4548
1152/1563 [=====================>........] - ETA: 0s - loss: 1.5433 - accuracy: 0.4555
1190/1563 [=====================>........] - ETA: 0s - loss: 1.5430 - accuracy: 0.4553
1228/1563 [======================>.......] - ETA: 0s - loss: 1.5429 - accuracy: 0.4554
1266/1563 [=======================>......] - ETA: 0s - loss: 1.5420 - accuracy: 0.4558
1304/1563 [========================>.....] - ETA: 0s - loss: 1.5404 - accuracy: 0.4565
1342/1563 [========================>.....] - ETA: 0s - loss: 1.5393 - accuracy: 0.4570
1380/1563 [=========================>....] - ETA: 0s - loss: 1.5391 - accuracy: 0.4567
1418/1563 [==========================>...] - ETA: 0s - loss: 1.5389 - accuracy: 0.4566
1456/1563 [==========================>...] - ETA: 0s - loss: 1.5393 - accuracy: 0.4560
1494/1563 [===========================>..] - ETA: 0s - loss: 1.5392 - accuracy: 0.4563
1532/1563 [============================>.] - ETA: 0s - loss: 1.5387 - accuracy: 0.4568
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5384 - accuracy: 0.4569
Epoch 4/10
1/1563 [..............................] - ETA: 2s - loss: 1.4693 - accuracy: 0.4062
39/1563 [..............................] - ETA: 2s - loss: 1.4932 - accuracy: 0.4696
77/1563 [>.............................] - ETA: 1s - loss: 1.4944 - accuracy: 0.4765
115/1563 [=>............................] - ETA: 1s - loss: 1.4861 - accuracy: 0.4793
153/1563 [=>............................] - ETA: 1s - loss: 1.4909 - accuracy: 0.4741
191/1563 [==>...........................] - ETA: 1s - loss: 1.4977 - accuracy: 0.4704
229/1563 [===>..........................] - ETA: 1s - loss: 1.4968 - accuracy: 0.4730
268/1563 [====>.........................] - ETA: 1s - loss: 1.4981 - accuracy: 0.4739
307/1563 [====>.........................] - ETA: 1s - loss: 1.4959 - accuracy: 0.4758
345/1563 [=====>........................] - ETA: 1s - loss: 1.4920 - accuracy: 0.4764
384/1563 [======>.......................] - ETA: 1s - loss: 1.4925 - accuracy: 0.4766
423/1563 [=======>......................] - ETA: 1s - loss: 1.4934 - accuracy: 0.4771
462/1563 [=======>......................] - ETA: 1s - loss: 1.4942 - accuracy: 0.4776
501/1563 [========>.....................] - ETA: 1s - loss: 1.4954 - accuracy: 0.4767
539/1563 [=========>....................] - ETA: 1s - loss: 1.4944 - accuracy: 0.4769
578/1563 [==========>...................] - ETA: 1s - loss: 1.4904 - accuracy: 0.4780
616/1563 [==========>...................] - ETA: 1s - loss: 1.4899 - accuracy: 0.4784
654/1563 [===========>..................] - ETA: 1s - loss: 1.4923 - accuracy: 0.4766
692/1563 [============>.................] - ETA: 1s - loss: 1.4920 - accuracy: 0.4764
731/1563 [=============>................] - ETA: 1s - loss: 1.4910 - accuracy: 0.4770
770/1563 [=============>................] - ETA: 1s - loss: 1.4902 - accuracy: 0.4772
809/1563 [==============>...............] - ETA: 0s - loss: 1.4906 - accuracy: 0.4765
848/1563 [===============>..............] - ETA: 0s - loss: 1.4906 - accuracy: 0.4761
887/1563 [================>.............] - ETA: 0s - loss: 1.4910 - accuracy: 0.4759
926/1563 [================>.............] - ETA: 0s - loss: 1.4891 - accuracy: 0.4764
965/1563 [=================>............] - ETA: 0s - loss: 1.4877 - accuracy: 0.4770
1003/1563 [==================>...........] - ETA: 0s - loss: 1.4876 - accuracy: 0.4776
1042/1563 [===================>..........] - ETA: 0s - loss: 1.4861 - accuracy: 0.4782
1081/1563 [===================>..........] - ETA: 0s - loss: 1.4846 - accuracy: 0.4781
1120/1563 [====================>.........] - ETA: 0s - loss: 1.4848 - accuracy: 0.4777
1159/1563 [=====================>........] - ETA: 0s - loss: 1.4847 - accuracy: 0.4779
1197/1563 [=====================>........] - ETA: 0s - loss: 1.4838 - accuracy: 0.4783
1235/1563 [======================>.......] - ETA: 0s - loss: 1.4834 - accuracy: 0.4783
1273/1563 [=======================>......] - ETA: 0s - loss: 1.4824 - accuracy: 0.4784
1311/1563 [========================>.....] - ETA: 0s - loss: 1.4810 - accuracy: 0.4790
1349/1563 [========================>.....] - ETA: 0s - loss: 1.4801 - accuracy: 0.4791
1388/1563 [=========================>....] - ETA: 0s - loss: 1.4794 - accuracy: 0.4791
1426/1563 [==========================>...] - ETA: 0s - loss: 1.4793 - accuracy: 0.4789
1464/1563 [===========================>..] - ETA: 0s - loss: 1.4793 - accuracy: 0.4790
1502/1563 [===========================>..] - ETA: 0s - loss: 1.4785 - accuracy: 0.4790
1540/1563 [============================>.] - ETA: 0s - loss: 1.4783 - accuracy: 0.4796
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4785 - accuracy: 0.4796
Epoch 5/10
1/1563 [..............................] - ETA: 2s - loss: 1.5355 - accuracy: 0.5000
39/1563 [..............................] - ETA: 2s - loss: 1.4489 - accuracy: 0.4752
77/1563 [>.............................] - ETA: 1s - loss: 1.4477 - accuracy: 0.4785
116/1563 [=>............................] - ETA: 1s - loss: 1.4301 - accuracy: 0.4844
155/1563 [=>............................] - ETA: 1s - loss: 1.4253 - accuracy: 0.4877
193/1563 [==>...........................] - ETA: 1s - loss: 1.4286 - accuracy: 0.4892
231/1563 [===>..........................] - ETA: 1s - loss: 1.4309 - accuracy: 0.4884
270/1563 [====>.........................] - ETA: 1s - loss: 1.4318 - accuracy: 0.4895
309/1563 [====>.........................] - ETA: 1s - loss: 1.4378 - accuracy: 0.4879
347/1563 [=====>........................] - ETA: 1s - loss: 1.4401 - accuracy: 0.4871
385/1563 [======>.......................] - ETA: 1s - loss: 1.4373 - accuracy: 0.4870
424/1563 [=======>......................] - ETA: 1s - loss: 1.4377 - accuracy: 0.4874
463/1563 [=======>......................] - ETA: 1s - loss: 1.4366 - accuracy: 0.4876
502/1563 [========>.....................] - ETA: 1s - loss: 1.4370 - accuracy: 0.4872
540/1563 [=========>....................] - ETA: 1s - loss: 1.4379 - accuracy: 0.4877
578/1563 [==========>...................] - ETA: 1s - loss: 1.4402 - accuracy: 0.4872
616/1563 [==========>...................] - ETA: 1s - loss: 1.4374 - accuracy: 0.4883
654/1563 [===========>..................] - ETA: 1s - loss: 1.4378 - accuracy: 0.4875
692/1563 [============>.................] - ETA: 1s - loss: 1.4353 - accuracy: 0.4888
730/1563 [=============>................] - ETA: 1s - loss: 1.4382 - accuracy: 0.4882
768/1563 [=============>................] - ETA: 1s - loss: 1.4370 - accuracy: 0.4891
806/1563 [==============>...............] - ETA: 1s - loss: 1.4365 - accuracy: 0.4899
844/1563 [===============>..............] - ETA: 0s - loss: 1.4364 - accuracy: 0.4909
882/1563 [===============>..............] - ETA: 0s - loss: 1.4350 - accuracy: 0.4916
920/1563 [================>.............] - ETA: 0s - loss: 1.4355 - accuracy: 0.4911
958/1563 [=================>............] - ETA: 0s - loss: 1.4354 - accuracy: 0.4915
996/1563 [==================>...........] - ETA: 0s - loss: 1.4338 - accuracy: 0.4922
1034/1563 [==================>...........] - ETA: 0s - loss: 1.4323 - accuracy: 0.4927
1072/1563 [===================>..........] - ETA: 0s - loss: 1.4301 - accuracy: 0.4939
1111/1563 [====================>.........] - ETA: 0s - loss: 1.4323 - accuracy: 0.4936
1149/1563 [=====================>........] - ETA: 0s - loss: 1.4324 - accuracy: 0.4938
1188/1563 [=====================>........] - ETA: 0s - loss: 1.4315 - accuracy: 0.4946
1226/1563 [======================>.......] - ETA: 0s - loss: 1.4309 - accuracy: 0.4954
1264/1563 [=======================>......] - ETA: 0s - loss: 1.4304 - accuracy: 0.4956
1302/1563 [=======================>......] - ETA: 0s - loss: 1.4315 - accuracy: 0.4955
1340/1563 [========================>.....] - ETA: 0s - loss: 1.4326 - accuracy: 0.4954
1378/1563 [=========================>....] - ETA: 0s - loss: 1.4327 - accuracy: 0.4954
1416/1563 [==========================>...] - ETA: 0s - loss: 1.4317 - accuracy: 0.4961
1454/1563 [==========================>...] - ETA: 0s - loss: 1.4319 - accuracy: 0.4960
1492/1563 [===========================>..] - ETA: 0s - loss: 1.4314 - accuracy: 0.4959
1530/1563 [============================>.] - ETA: 0s - loss: 1.4313 - accuracy: 0.4957
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4300 - accuracy: 0.4962
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.2415 - accuracy: 0.6562
39/1563 [..............................] - ETA: 2s - loss: 1.3854 - accuracy: 0.5208
77/1563 [>.............................] - ETA: 1s - loss: 1.3960 - accuracy: 0.5219
115/1563 [=>............................] - ETA: 1s - loss: 1.3825 - accuracy: 0.5236
153/1563 [=>............................] - ETA: 1s - loss: 1.3873 - accuracy: 0.5157
192/1563 [==>...........................] - ETA: 1s - loss: 1.3905 - accuracy: 0.5125
231/1563 [===>..........................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5091
269/1563 [====>.........................] - ETA: 1s - loss: 1.3848 - accuracy: 0.5148
307/1563 [====>.........................] - ETA: 1s - loss: 1.3906 - accuracy: 0.5150
346/1563 [=====>........................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5127
384/1563 [======>.......................] - ETA: 1s - loss: 1.3862 - accuracy: 0.5161
422/1563 [=======>......................] - ETA: 1s - loss: 1.3926 - accuracy: 0.5124
461/1563 [=======>......................] - ETA: 1s - loss: 1.3956 - accuracy: 0.5118
500/1563 [========>.....................] - ETA: 1s - loss: 1.3964 - accuracy: 0.5122
539/1563 [=========>....................] - ETA: 1s - loss: 1.3936 - accuracy: 0.5123
577/1563 [==========>...................] - ETA: 1s - loss: 1.3934 - accuracy: 0.5128
615/1563 [==========>...................] - ETA: 1s - loss: 1.3952 - accuracy: 0.5113
653/1563 [===========>..................] - ETA: 1s - loss: 1.3950 - accuracy: 0.5117
691/1563 [============>.................] - ETA: 1s - loss: 1.3908 - accuracy: 0.5132
729/1563 [============>.................] - ETA: 1s - loss: 1.3900 - accuracy: 0.5140
768/1563 [=============>................] - ETA: 1s - loss: 1.3927 - accuracy: 0.5127
807/1563 [==============>...............] - ETA: 1s - loss: 1.3928 - accuracy: 0.5136
846/1563 [===============>..............] - ETA: 0s - loss: 1.3913 - accuracy: 0.5141
884/1563 [===============>..............] - ETA: 0s - loss: 1.3896 - accuracy: 0.5146
922/1563 [================>.............] - ETA: 0s - loss: 1.3892 - accuracy: 0.5148
960/1563 [=================>............] - ETA: 0s - loss: 1.3909 - accuracy: 0.5137
998/1563 [==================>...........] - ETA: 0s - loss: 1.3901 - accuracy: 0.5134
1036/1563 [==================>...........] - ETA: 0s - loss: 1.3913 - accuracy: 0.5121
1074/1563 [===================>..........] - ETA: 0s - loss: 1.3888 - accuracy: 0.5129
1112/1563 [====================>.........] - ETA: 0s - loss: 1.3891 - accuracy: 0.5123
1150/1563 [=====================>........] - ETA: 0s - loss: 1.3893 - accuracy: 0.5123
1188/1563 [=====================>........] - ETA: 0s - loss: 1.3909 - accuracy: 0.5120
1226/1563 [======================>.......] - ETA: 0s - loss: 1.3898 - accuracy: 0.5128
1264/1563 [=======================>......] - ETA: 0s - loss: 1.3895 - accuracy: 0.5132
1302/1563 [=======================>......] - ETA: 0s - loss: 1.3895 - accuracy: 0.5132
1341/1563 [========================>.....] - ETA: 0s - loss: 1.3909 - accuracy: 0.5124
1380/1563 [=========================>....] - ETA: 0s - loss: 1.3905 - accuracy: 0.5123
1419/1563 [==========================>...] - ETA: 0s - loss: 1.3891 - accuracy: 0.5131
1457/1563 [==========================>...] - ETA: 0s - loss: 1.3874 - accuracy: 0.5138
1495/1563 [===========================>..] - ETA: 0s - loss: 1.3880 - accuracy: 0.5134
1533/1563 [============================>.] - ETA: 0s - loss: 1.3879 - accuracy: 0.5136
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3883 - accuracy: 0.5138
Epoch 7/10
1/1563 [..............................] - ETA: 2s - loss: 1.1637 - accuracy: 0.5625
39/1563 [..............................] - ETA: 2s - loss: 1.3064 - accuracy: 0.5457
78/1563 [>.............................] - ETA: 1s - loss: 1.3357 - accuracy: 0.5292
116/1563 [=>............................] - ETA: 1s - loss: 1.3453 - accuracy: 0.5294
155/1563 [=>............................] - ETA: 1s - loss: 1.3454 - accuracy: 0.5292
193/1563 [==>...........................] - ETA: 1s - loss: 1.3511 - accuracy: 0.5270
231/1563 [===>..........................] - ETA: 1s - loss: 1.3588 - accuracy: 0.5218
269/1563 [====>.........................] - ETA: 1s - loss: 1.3637 - accuracy: 0.5206
307/1563 [====>.........................] - ETA: 1s - loss: 1.3608 - accuracy: 0.5237
345/1563 [=====>........................] - ETA: 1s - loss: 1.3621 - accuracy: 0.5235
383/1563 [======>.......................] - ETA: 1s - loss: 1.3580 - accuracy: 0.5252
421/1563 [=======>......................] - ETA: 1s - loss: 1.3616 - accuracy: 0.5242
460/1563 [=======>......................] - ETA: 1s - loss: 1.3642 - accuracy: 0.5220
498/1563 [========>.....................] - ETA: 1s - loss: 1.3622 - accuracy: 0.5208
536/1563 [=========>....................] - ETA: 1s - loss: 1.3602 - accuracy: 0.5227
574/1563 [==========>...................] - ETA: 1s - loss: 1.3564 - accuracy: 0.5226
612/1563 [==========>...................] - ETA: 1s - loss: 1.3572 - accuracy: 0.5221
650/1563 [===========>..................] - ETA: 1s - loss: 1.3579 - accuracy: 0.5225
688/1563 [============>.................] - ETA: 1s - loss: 1.3559 - accuracy: 0.5231
726/1563 [============>.................] - ETA: 1s - loss: 1.3550 - accuracy: 0.5241
764/1563 [=============>................] - ETA: 1s - loss: 1.3538 - accuracy: 0.5243
802/1563 [==============>...............] - ETA: 1s - loss: 1.3543 - accuracy: 0.5245
840/1563 [===============>..............] - ETA: 0s - loss: 1.3536 - accuracy: 0.5249
878/1563 [===============>..............] - ETA: 0s - loss: 1.3550 - accuracy: 0.5247
916/1563 [================>.............] - ETA: 0s - loss: 1.3537 - accuracy: 0.5252
954/1563 [=================>............] - ETA: 0s - loss: 1.3522 - accuracy: 0.5257
992/1563 [==================>...........] - ETA: 0s - loss: 1.3529 - accuracy: 0.5258
1030/1563 [==================>...........] - ETA: 0s - loss: 1.3522 - accuracy: 0.5262
1068/1563 [===================>..........] - ETA: 0s - loss: 1.3512 - accuracy: 0.5263
1106/1563 [====================>.........] - ETA: 0s - loss: 1.3519 - accuracy: 0.5261
1144/1563 [====================>.........] - ETA: 0s - loss: 1.3508 - accuracy: 0.5271
1182/1563 [=====================>........] - ETA: 0s - loss: 1.3513 - accuracy: 0.5268
1220/1563 [======================>.......] - ETA: 0s - loss: 1.3515 - accuracy: 0.5269
1258/1563 [=======================>......] - ETA: 0s - loss: 1.3515 - accuracy: 0.5264
1296/1563 [=======================>......] - ETA: 0s - loss: 1.3528 - accuracy: 0.5260
1334/1563 [========================>.....] - ETA: 0s - loss: 1.3516 - accuracy: 0.5266
1372/1563 [=========================>....] - ETA: 0s - loss: 1.3495 - accuracy: 0.5267
1410/1563 [==========================>...] - ETA: 0s - loss: 1.3484 - accuracy: 0.5273
1448/1563 [==========================>...] - ETA: 0s - loss: 1.3484 - accuracy: 0.5271
1486/1563 [===========================>..] - ETA: 0s - loss: 1.3487 - accuracy: 0.5272
1524/1563 [============================>.] - ETA: 0s - loss: 1.3500 - accuracy: 0.5266
1562/1563 [============================>.] - ETA: 0s - loss: 1.3495 - accuracy: 0.5265
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3494 - accuracy: 0.5266
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.2294 - accuracy: 0.4688
39/1563 [..............................] - ETA: 2s - loss: 1.3081 - accuracy: 0.5337
77/1563 [>.............................] - ETA: 1s - loss: 1.3150 - accuracy: 0.5349
115/1563 [=>............................] - ETA: 1s - loss: 1.3030 - accuracy: 0.5386
153/1563 [=>............................] - ETA: 1s - loss: 1.3093 - accuracy: 0.5398
191/1563 [==>...........................] - ETA: 1s - loss: 1.3121 - accuracy: 0.5411
229/1563 [===>..........................] - ETA: 1s - loss: 1.3117 - accuracy: 0.5405
267/1563 [====>.........................] - ETA: 1s - loss: 1.3035 - accuracy: 0.5439
305/1563 [====>.........................] - ETA: 1s - loss: 1.3058 - accuracy: 0.5433
343/1563 [=====>........................] - ETA: 1s - loss: 1.3115 - accuracy: 0.5410
382/1563 [======>.......................] - ETA: 1s - loss: 1.3140 - accuracy: 0.5398
420/1563 [=======>......................] - ETA: 1s - loss: 1.3109 - accuracy: 0.5401
459/1563 [=======>......................] - ETA: 1s - loss: 1.3076 - accuracy: 0.5400
497/1563 [========>.....................] - ETA: 1s - loss: 1.3118 - accuracy: 0.5390
536/1563 [=========>....................] - ETA: 1s - loss: 1.3104 - accuracy: 0.5400
574/1563 [==========>...................] - ETA: 1s - loss: 1.3135 - accuracy: 0.5401
612/1563 [==========>...................] - ETA: 1s - loss: 1.3143 - accuracy: 0.5394
650/1563 [===========>..................] - ETA: 1s - loss: 1.3133 - accuracy: 0.5402
688/1563 [============>.................] - ETA: 1s - loss: 1.3114 - accuracy: 0.5408
726/1563 [============>.................] - ETA: 1s - loss: 1.3126 - accuracy: 0.5400
765/1563 [=============>................] - ETA: 1s - loss: 1.3129 - accuracy: 0.5402
803/1563 [==============>...............] - ETA: 1s - loss: 1.3127 - accuracy: 0.5411
841/1563 [===============>..............] - ETA: 0s - loss: 1.3107 - accuracy: 0.5417
879/1563 [===============>..............] - ETA: 0s - loss: 1.3102 - accuracy: 0.5426
917/1563 [================>.............] - ETA: 0s - loss: 1.3140 - accuracy: 0.5402
955/1563 [=================>............] - ETA: 0s - loss: 1.3133 - accuracy: 0.5402
993/1563 [==================>...........] - ETA: 0s - loss: 1.3138 - accuracy: 0.5396
1032/1563 [==================>...........] - ETA: 0s - loss: 1.3121 - accuracy: 0.5402
1070/1563 [===================>..........] - ETA: 0s - loss: 1.3121 - accuracy: 0.5402
1108/1563 [====================>.........] - ETA: 0s - loss: 1.3123 - accuracy: 0.5401
1146/1563 [====================>.........] - ETA: 0s - loss: 1.3136 - accuracy: 0.5402
1184/1563 [=====================>........] - ETA: 0s - loss: 1.3128 - accuracy: 0.5401
1222/1563 [======================>.......] - ETA: 0s - loss: 1.3138 - accuracy: 0.5403
1260/1563 [=======================>......] - ETA: 0s - loss: 1.3138 - accuracy: 0.5403
1298/1563 [=======================>......] - ETA: 0s - loss: 1.3155 - accuracy: 0.5396
1336/1563 [========================>.....] - ETA: 0s - loss: 1.3155 - accuracy: 0.5392
1374/1563 [=========================>....] - ETA: 0s - loss: 1.3152 - accuracy: 0.5393
1412/1563 [==========================>...] - ETA: 0s - loss: 1.3142 - accuracy: 0.5396
1450/1563 [==========================>...] - ETA: 0s - loss: 1.3151 - accuracy: 0.5392
1488/1563 [===========================>..] - ETA: 0s - loss: 1.3143 - accuracy: 0.5397
1526/1563 [============================>.] - ETA: 0s - loss: 1.3148 - accuracy: 0.5398
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3152 - accuracy: 0.5396
Epoch 9/10
1/1563 [..............................] - ETA: 2s - loss: 1.1669 - accuracy: 0.5938
39/1563 [..............................] - ETA: 2s - loss: 1.2332 - accuracy: 0.5593
78/1563 [>.............................] - ETA: 1s - loss: 1.2554 - accuracy: 0.5545
117/1563 [=>............................] - ETA: 1s - loss: 1.2393 - accuracy: 0.5609
156/1563 [=>............................] - ETA: 1s - loss: 1.2468 - accuracy: 0.5605
195/1563 [==>...........................] - ETA: 1s - loss: 1.2524 - accuracy: 0.5612
233/1563 [===>..........................] - ETA: 1s - loss: 1.2616 - accuracy: 0.5589
271/1563 [====>.........................] - ETA: 1s - loss: 1.2592 - accuracy: 0.5589
309/1563 [====>.........................] - ETA: 1s - loss: 1.2598 - accuracy: 0.5597
347/1563 [=====>........................] - ETA: 1s - loss: 1.2652 - accuracy: 0.5575
385/1563 [======>.......................] - ETA: 1s - loss: 1.2688 - accuracy: 0.5555
423/1563 [=======>......................] - ETA: 1s - loss: 1.2668 - accuracy: 0.5563
461/1563 [=======>......................] - ETA: 1s - loss: 1.2694 - accuracy: 0.5563
499/1563 [========>.....................] - ETA: 1s - loss: 1.2706 - accuracy: 0.5554
537/1563 [=========>....................] - ETA: 1s - loss: 1.2740 - accuracy: 0.5545
575/1563 [==========>...................] - ETA: 1s - loss: 1.2733 - accuracy: 0.5560
613/1563 [==========>...................] - ETA: 1s - loss: 1.2726 - accuracy: 0.5565
651/1563 [===========>..................] - ETA: 1s - loss: 1.2711 - accuracy: 0.5573
689/1563 [============>.................] - ETA: 1s - loss: 1.2718 - accuracy: 0.5567
727/1563 [============>.................] - ETA: 1s - loss: 1.2745 - accuracy: 0.5560
765/1563 [=============>................] - ETA: 1s - loss: 1.2783 - accuracy: 0.5545
803/1563 [==============>...............] - ETA: 1s - loss: 1.2796 - accuracy: 0.5535
842/1563 [===============>..............] - ETA: 0s - loss: 1.2781 - accuracy: 0.5539
880/1563 [===============>..............] - ETA: 0s - loss: 1.2771 - accuracy: 0.5544
919/1563 [================>.............] - ETA: 0s - loss: 1.2771 - accuracy: 0.5536
958/1563 [=================>............] - ETA: 0s - loss: 1.2779 - accuracy: 0.5535
996/1563 [==================>...........] - ETA: 0s - loss: 1.2804 - accuracy: 0.5529
1034/1563 [==================>...........] - ETA: 0s - loss: 1.2795 - accuracy: 0.5533
1072/1563 [===================>..........] - ETA: 0s - loss: 1.2817 - accuracy: 0.5522
1110/1563 [====================>.........] - ETA: 0s - loss: 1.2813 - accuracy: 0.5518
1148/1563 [=====================>........] - ETA: 0s - loss: 1.2821 - accuracy: 0.5515
1186/1563 [=====================>........] - ETA: 0s - loss: 1.2834 - accuracy: 0.5511
1225/1563 [======================>.......] - ETA: 0s - loss: 1.2827 - accuracy: 0.5516
1263/1563 [=======================>......] - ETA: 0s - loss: 1.2825 - accuracy: 0.5516
1302/1563 [=======================>......] - ETA: 0s - loss: 1.2819 - accuracy: 0.5518
1341/1563 [========================>.....] - ETA: 0s - loss: 1.2817 - accuracy: 0.5517
1379/1563 [=========================>....] - ETA: 0s - loss: 1.2816 - accuracy: 0.5517
1417/1563 [==========================>...] - ETA: 0s - loss: 1.2811 - accuracy: 0.5515
1455/1563 [==========================>...] - ETA: 0s - loss: 1.2825 - accuracy: 0.5506
1493/1563 [===========================>..] - ETA: 0s - loss: 1.2829 - accuracy: 0.5503
1531/1563 [============================>.] - ETA: 0s - loss: 1.2830 - accuracy: 0.5507
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2837 - accuracy: 0.5505
Epoch 10/10
1/1563 [..............................] - ETA: 2s - loss: 1.0024 - accuracy: 0.6875
39/1563 [..............................] - ETA: 2s - loss: 1.2177 - accuracy: 0.5681
77/1563 [>.............................] - ETA: 1s - loss: 1.2453 - accuracy: 0.5637
115/1563 [=>............................] - ETA: 1s - loss: 1.2528 - accuracy: 0.5560
152/1563 [=>............................] - ETA: 1s - loss: 1.2557 - accuracy: 0.5576
190/1563 [==>...........................] - ETA: 1s - loss: 1.2622 - accuracy: 0.5574
228/1563 [===>..........................] - ETA: 1s - loss: 1.2685 - accuracy: 0.5533
266/1563 [====>.........................] - ETA: 1s - loss: 1.2658 - accuracy: 0.5519
304/1563 [====>.........................] - ETA: 1s - loss: 1.2643 - accuracy: 0.5538
342/1563 [=====>........................] - ETA: 1s - loss: 1.2612 - accuracy: 0.5570
380/1563 [======>.......................] - ETA: 1s - loss: 1.2600 - accuracy: 0.5579
418/1563 [=======>......................] - ETA: 1s - loss: 1.2597 - accuracy: 0.5583
456/1563 [=======>......................] - ETA: 1s - loss: 1.2593 - accuracy: 0.5578
494/1563 [========>.....................] - ETA: 1s - loss: 1.2554 - accuracy: 0.5600
532/1563 [=========>....................] - ETA: 1s - loss: 1.2553 - accuracy: 0.5586
570/1563 [=========>....................] - ETA: 1s - loss: 1.2528 - accuracy: 0.5594
608/1563 [==========>...................] - ETA: 1s - loss: 1.2546 - accuracy: 0.5589
646/1563 [===========>..................] - ETA: 1s - loss: 1.2538 - accuracy: 0.5586
684/1563 [============>.................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5584
722/1563 [============>.................] - ETA: 1s - loss: 1.2542 - accuracy: 0.5582
760/1563 [=============>................] - ETA: 1s - loss: 1.2520 - accuracy: 0.5582
799/1563 [==============>...............] - ETA: 1s - loss: 1.2542 - accuracy: 0.5577
838/1563 [===============>..............] - ETA: 0s - loss: 1.2568 - accuracy: 0.5564
876/1563 [===============>..............] - ETA: 0s - loss: 1.2588 - accuracy: 0.5560
914/1563 [================>.............] - ETA: 0s - loss: 1.2577 - accuracy: 0.5563
952/1563 [=================>............] - ETA: 0s - loss: 1.2584 - accuracy: 0.5564
990/1563 [==================>...........] - ETA: 0s - loss: 1.2568 - accuracy: 0.5569
1028/1563 [==================>...........] - ETA: 0s - loss: 1.2548 - accuracy: 0.5579
1066/1563 [===================>..........] - ETA: 0s - loss: 1.2534 - accuracy: 0.5588
1104/1563 [====================>.........] - ETA: 0s - loss: 1.2532 - accuracy: 0.5594
1142/1563 [====================>.........] - ETA: 0s - loss: 1.2542 - accuracy: 0.5591
1180/1563 [=====================>........] - ETA: 0s - loss: 1.2570 - accuracy: 0.5578
1217/1563 [======================>.......] - ETA: 0s - loss: 1.2561 - accuracy: 0.5583
1255/1563 [=======================>......] - ETA: 0s - loss: 1.2566 - accuracy: 0.5582
1293/1563 [=======================>......] - ETA: 0s - loss: 1.2567 - accuracy: 0.5584
1331/1563 [========================>.....] - ETA: 0s - loss: 1.2558 - accuracy: 0.5585
1369/1563 [=========================>....] - ETA: 0s - loss: 1.2557 - accuracy: 0.5586
1407/1563 [==========================>...] - ETA: 0s - loss: 1.2553 - accuracy: 0.5588
1445/1563 [==========================>...] - ETA: 0s - loss: 1.2559 - accuracy: 0.5586
1483/1563 [===========================>..] - ETA: 0s - loss: 1.2558 - accuracy: 0.5586
1521/1563 [============================>.] - ETA: 0s - loss: 1.2552 - accuracy: 0.5587
1559/1563 [============================>.] - ETA: 0s - loss: 1.2550 - accuracy: 0.5588
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2551 - accuracy: 0.5589
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 38.72656440734863
hydro04
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins17753532866450384906.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
