Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 96747 queued and waiting for resources
srun: job 96747 has been allocated resources
2024-11-05 07:43:06.779463: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-05 07:43:28.496641: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-05 07:43:58.713108: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-05 07:43:58.713663: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-05 07:43:58.713678: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-11-05 07:44:43.394703: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-05 07:44:45.846660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-11-05 07:44:53.689130: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 1:44:44 - loss: 2.3815 - accuracy: 0.0312
38/1563 [..............................] - ETA: 2s - loss: 2.2483 - accuracy: 0.1653
77/1563 [>.............................] - ETA: 1s - loss: 2.1781 - accuracy: 0.1936
116/1563 [=>............................] - ETA: 1s - loss: 2.1220 - accuracy: 0.2214
155/1563 [=>............................] - ETA: 1s - loss: 2.0914 - accuracy: 0.2361
194/1563 [==>...........................] - ETA: 1s - loss: 2.0632 - accuracy: 0.2492
233/1563 [===>..........................] - ETA: 1s - loss: 2.0433 - accuracy: 0.2546
271/1563 [====>.........................] - ETA: 1s - loss: 2.0210 - accuracy: 0.2650
310/1563 [====>.........................] - ETA: 1s - loss: 2.0056 - accuracy: 0.2711
349/1563 [=====>........................] - ETA: 1s - loss: 1.9936 - accuracy: 0.2783
388/1563 [======>.......................] - ETA: 1s - loss: 1.9784 - accuracy: 0.2841
427/1563 [=======>......................] - ETA: 1s - loss: 1.9678 - accuracy: 0.2900
466/1563 [=======>......................] - ETA: 1s - loss: 1.9538 - accuracy: 0.2959
506/1563 [========>.....................] - ETA: 1s - loss: 1.9429 - accuracy: 0.2998
545/1563 [=========>....................] - ETA: 1s - loss: 1.9339 - accuracy: 0.3035
584/1563 [==========>...................] - ETA: 1s - loss: 1.9248 - accuracy: 0.3089
624/1563 [==========>...................] - ETA: 1s - loss: 1.9145 - accuracy: 0.3126
664/1563 [===========>..................] - ETA: 1s - loss: 1.9079 - accuracy: 0.3149
703/1563 [============>.................] - ETA: 1s - loss: 1.8989 - accuracy: 0.3185
743/1563 [=============>................] - ETA: 1s - loss: 1.8937 - accuracy: 0.3208
782/1563 [==============>...............] - ETA: 1s - loss: 1.8890 - accuracy: 0.3223
821/1563 [==============>...............] - ETA: 0s - loss: 1.8851 - accuracy: 0.3250
861/1563 [===============>..............] - ETA: 0s - loss: 1.8794 - accuracy: 0.3272
900/1563 [================>.............] - ETA: 0s - loss: 1.8736 - accuracy: 0.3288
939/1563 [=================>............] - ETA: 0s - loss: 1.8690 - accuracy: 0.3307
978/1563 [=================>............] - ETA: 0s - loss: 1.8625 - accuracy: 0.3338
1017/1563 [==================>...........] - ETA: 0s - loss: 1.8574 - accuracy: 0.3366
1056/1563 [===================>..........] - ETA: 0s - loss: 1.8533 - accuracy: 0.3376
1096/1563 [====================>.........] - ETA: 0s - loss: 1.8485 - accuracy: 0.3393
1136/1563 [====================>.........] - ETA: 0s - loss: 1.8443 - accuracy: 0.3412
1175/1563 [=====================>........] - ETA: 0s - loss: 1.8399 - accuracy: 0.3433
1215/1563 [======================>.......] - ETA: 0s - loss: 1.8381 - accuracy: 0.3442
1255/1563 [=======================>......] - ETA: 0s - loss: 1.8342 - accuracy: 0.3458
1295/1563 [=======================>......] - ETA: 0s - loss: 1.8308 - accuracy: 0.3472
1335/1563 [========================>.....] - ETA: 0s - loss: 1.8278 - accuracy: 0.3485
1375/1563 [=========================>....] - ETA: 0s - loss: 1.8243 - accuracy: 0.3499
1415/1563 [==========================>...] - ETA: 0s - loss: 1.8220 - accuracy: 0.3511
1454/1563 [==========================>...] - ETA: 0s - loss: 1.8182 - accuracy: 0.3529
1494/1563 [===========================>..] - ETA: 0s - loss: 1.8150 - accuracy: 0.3539
1533/1563 [============================>.] - ETA: 0s - loss: 1.8123 - accuracy: 0.3546
1563/1563 [==============================] - 6s 1ms/step - loss: 1.8106 - accuracy: 0.3550
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 1.4349 - accuracy: 0.5000
41/1563 [..............................] - ETA: 1s - loss: 1.6196 - accuracy: 0.4405
81/1563 [>.............................] - ETA: 1s - loss: 1.6565 - accuracy: 0.4232
121/1563 [=>............................] - ETA: 1s - loss: 1.6517 - accuracy: 0.4192
161/1563 [==>...........................] - ETA: 1s - loss: 1.6585 - accuracy: 0.4140
201/1563 [==>...........................] - ETA: 1s - loss: 1.6583 - accuracy: 0.4109
241/1563 [===>..........................] - ETA: 1s - loss: 1.6557 - accuracy: 0.4109
280/1563 [====>.........................] - ETA: 1s - loss: 1.6479 - accuracy: 0.4134
320/1563 [=====>........................] - ETA: 1s - loss: 1.6447 - accuracy: 0.4159
360/1563 [=====>........................] - ETA: 1s - loss: 1.6451 - accuracy: 0.4156
400/1563 [======>.......................] - ETA: 1s - loss: 1.6442 - accuracy: 0.4155
440/1563 [=======>......................] - ETA: 1s - loss: 1.6441 - accuracy: 0.4170
480/1563 [========>.....................] - ETA: 1s - loss: 1.6455 - accuracy: 0.4178
520/1563 [========>.....................] - ETA: 1s - loss: 1.6464 - accuracy: 0.4175
560/1563 [=========>....................] - ETA: 1s - loss: 1.6475 - accuracy: 0.4181
600/1563 [==========>...................] - ETA: 1s - loss: 1.6466 - accuracy: 0.4182
640/1563 [===========>..................] - ETA: 1s - loss: 1.6428 - accuracy: 0.4198
680/1563 [============>.................] - ETA: 1s - loss: 1.6434 - accuracy: 0.4197
720/1563 [============>.................] - ETA: 1s - loss: 1.6425 - accuracy: 0.4195
760/1563 [=============>................] - ETA: 1s - loss: 1.6409 - accuracy: 0.4201
800/1563 [==============>...............] - ETA: 0s - loss: 1.6395 - accuracy: 0.4200
840/1563 [===============>..............] - ETA: 0s - loss: 1.6376 - accuracy: 0.4208
880/1563 [===============>..............] - ETA: 0s - loss: 1.6364 - accuracy: 0.4199
920/1563 [================>.............] - ETA: 0s - loss: 1.6370 - accuracy: 0.4208
960/1563 [=================>............] - ETA: 0s - loss: 1.6373 - accuracy: 0.4210
1000/1563 [==================>...........] - ETA: 0s - loss: 1.6361 - accuracy: 0.4217
1040/1563 [==================>...........] - ETA: 0s - loss: 1.6352 - accuracy: 0.4219
1080/1563 [===================>..........] - ETA: 0s - loss: 1.6331 - accuracy: 0.4231
1119/1563 [====================>.........] - ETA: 0s - loss: 1.6322 - accuracy: 0.4238
1158/1563 [=====================>........] - ETA: 0s - loss: 1.6319 - accuracy: 0.4233
1198/1563 [=====================>........] - ETA: 0s - loss: 1.6315 - accuracy: 0.4232
1237/1563 [======================>.......] - ETA: 0s - loss: 1.6306 - accuracy: 0.4235
1277/1563 [=======================>......] - ETA: 0s - loss: 1.6299 - accuracy: 0.4242
1317/1563 [========================>.....] - ETA: 0s - loss: 1.6298 - accuracy: 0.4245
1357/1563 [=========================>....] - ETA: 0s - loss: 1.6283 - accuracy: 0.4249
1397/1563 [=========================>....] - ETA: 0s - loss: 1.6270 - accuracy: 0.4254
1436/1563 [==========================>...] - ETA: 0s - loss: 1.6261 - accuracy: 0.4258
1475/1563 [===========================>..] - ETA: 0s - loss: 1.6247 - accuracy: 0.4259
1515/1563 [============================>.] - ETA: 0s - loss: 1.6239 - accuracy: 0.4263
1554/1563 [============================>.] - ETA: 0s - loss: 1.6224 - accuracy: 0.4267
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6223 - accuracy: 0.4268
Epoch 3/10
1/1563 [..............................] - ETA: 1s - loss: 1.4495 - accuracy: 0.4375
41/1563 [..............................] - ETA: 1s - loss: 1.5587 - accuracy: 0.4421
81/1563 [>.............................] - ETA: 1s - loss: 1.5533 - accuracy: 0.4452
121/1563 [=>............................] - ETA: 1s - loss: 1.5486 - accuracy: 0.4378
161/1563 [==>...........................] - ETA: 1s - loss: 1.5580 - accuracy: 0.4356
201/1563 [==>...........................] - ETA: 1s - loss: 1.5593 - accuracy: 0.4366
241/1563 [===>..........................] - ETA: 1s - loss: 1.5667 - accuracy: 0.4324
280/1563 [====>.........................] - ETA: 1s - loss: 1.5671 - accuracy: 0.4323
318/1563 [=====>........................] - ETA: 1s - loss: 1.5706 - accuracy: 0.4308
358/1563 [=====>........................] - ETA: 1s - loss: 1.5674 - accuracy: 0.4351
398/1563 [======>.......................] - ETA: 1s - loss: 1.5680 - accuracy: 0.4362
438/1563 [=======>......................] - ETA: 1s - loss: 1.5652 - accuracy: 0.4375
477/1563 [========>.....................] - ETA: 1s - loss: 1.5660 - accuracy: 0.4365
517/1563 [========>.....................] - ETA: 1s - loss: 1.5643 - accuracy: 0.4395
557/1563 [=========>....................] - ETA: 1s - loss: 1.5638 - accuracy: 0.4391
597/1563 [==========>...................] - ETA: 1s - loss: 1.5599 - accuracy: 0.4421
637/1563 [===========>..................] - ETA: 1s - loss: 1.5574 - accuracy: 0.4438
676/1563 [===========>..................] - ETA: 1s - loss: 1.5569 - accuracy: 0.4441
715/1563 [============>.................] - ETA: 1s - loss: 1.5550 - accuracy: 0.4461
755/1563 [=============>................] - ETA: 1s - loss: 1.5534 - accuracy: 0.4474
794/1563 [==============>...............] - ETA: 0s - loss: 1.5504 - accuracy: 0.4489
834/1563 [===============>..............] - ETA: 0s - loss: 1.5513 - accuracy: 0.4493
873/1563 [===============>..............] - ETA: 0s - loss: 1.5528 - accuracy: 0.4491
912/1563 [================>.............] - ETA: 0s - loss: 1.5529 - accuracy: 0.4492
951/1563 [=================>............] - ETA: 0s - loss: 1.5504 - accuracy: 0.4495
990/1563 [==================>...........] - ETA: 0s - loss: 1.5511 - accuracy: 0.4501
1029/1563 [==================>...........] - ETA: 0s - loss: 1.5499 - accuracy: 0.4507
1068/1563 [===================>..........] - ETA: 0s - loss: 1.5487 - accuracy: 0.4518
1107/1563 [====================>.........] - ETA: 0s - loss: 1.5479 - accuracy: 0.4527
1146/1563 [====================>.........] - ETA: 0s - loss: 1.5466 - accuracy: 0.4532
1185/1563 [=====================>........] - ETA: 0s - loss: 1.5450 - accuracy: 0.4536
1224/1563 [======================>.......] - ETA: 0s - loss: 1.5444 - accuracy: 0.4542
1263/1563 [=======================>......] - ETA: 0s - loss: 1.5417 - accuracy: 0.4552
1303/1563 [========================>.....] - ETA: 0s - loss: 1.5414 - accuracy: 0.4555
1342/1563 [========================>.....] - ETA: 0s - loss: 1.5419 - accuracy: 0.4555
1381/1563 [=========================>....] - ETA: 0s - loss: 1.5404 - accuracy: 0.4560
1420/1563 [==========================>...] - ETA: 0s - loss: 1.5406 - accuracy: 0.4563
1459/1563 [===========================>..] - ETA: 0s - loss: 1.5413 - accuracy: 0.4563
1497/1563 [===========================>..] - ETA: 0s - loss: 1.5406 - accuracy: 0.4564
1537/1563 [============================>.] - ETA: 0s - loss: 1.5407 - accuracy: 0.4562
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5416 - accuracy: 0.4562
Epoch 4/10
1/1563 [..............................] - ETA: 2s - loss: 1.3514 - accuracy: 0.5625
41/1563 [..............................] - ETA: 1s - loss: 1.5309 - accuracy: 0.4512
81/1563 [>.............................] - ETA: 1s - loss: 1.5130 - accuracy: 0.4633
121/1563 [=>............................] - ETA: 1s - loss: 1.4942 - accuracy: 0.4677
162/1563 [==>...........................] - ETA: 1s - loss: 1.4858 - accuracy: 0.4709
202/1563 [==>...........................] - ETA: 1s - loss: 1.4822 - accuracy: 0.4773
242/1563 [===>..........................] - ETA: 1s - loss: 1.4850 - accuracy: 0.4771
282/1563 [====>.........................] - ETA: 1s - loss: 1.4795 - accuracy: 0.4779
323/1563 [=====>........................] - ETA: 1s - loss: 1.4791 - accuracy: 0.4787
363/1563 [=====>........................] - ETA: 1s - loss: 1.4861 - accuracy: 0.4767
403/1563 [======>.......................] - ETA: 1s - loss: 1.4883 - accuracy: 0.4753
443/1563 [=======>......................] - ETA: 1s - loss: 1.4899 - accuracy: 0.4735
483/1563 [========>.....................] - ETA: 1s - loss: 1.4897 - accuracy: 0.4739
523/1563 [=========>....................] - ETA: 1s - loss: 1.4888 - accuracy: 0.4738
563/1563 [=========>....................] - ETA: 1s - loss: 1.4888 - accuracy: 0.4741
603/1563 [==========>...................] - ETA: 1s - loss: 1.4873 - accuracy: 0.4748
643/1563 [===========>..................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4740
684/1563 [============>.................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4736
724/1563 [============>.................] - ETA: 1s - loss: 1.4896 - accuracy: 0.4739
764/1563 [=============>................] - ETA: 1s - loss: 1.4900 - accuracy: 0.4741
804/1563 [==============>...............] - ETA: 0s - loss: 1.4910 - accuracy: 0.4738
844/1563 [===============>..............] - ETA: 0s - loss: 1.4936 - accuracy: 0.4723
885/1563 [===============>..............] - ETA: 0s - loss: 1.4906 - accuracy: 0.4737
926/1563 [================>.............] - ETA: 0s - loss: 1.4909 - accuracy: 0.4735
967/1563 [=================>............] - ETA: 0s - loss: 1.4893 - accuracy: 0.4740
1008/1563 [==================>...........] - ETA: 0s - loss: 1.4873 - accuracy: 0.4749
1049/1563 [===================>..........] - ETA: 0s - loss: 1.4850 - accuracy: 0.4758
1090/1563 [===================>..........] - ETA: 0s - loss: 1.4859 - accuracy: 0.4759
1130/1563 [====================>.........] - ETA: 0s - loss: 1.4855 - accuracy: 0.4762
1171/1563 [=====================>........] - ETA: 0s - loss: 1.4849 - accuracy: 0.4763
1211/1563 [======================>.......] - ETA: 0s - loss: 1.4842 - accuracy: 0.4771
1252/1563 [=======================>......] - ETA: 0s - loss: 1.4832 - accuracy: 0.4774
1292/1563 [=======================>......] - ETA: 0s - loss: 1.4820 - accuracy: 0.4779
1332/1563 [========================>.....] - ETA: 0s - loss: 1.4838 - accuracy: 0.4764
1372/1563 [=========================>....] - ETA: 0s - loss: 1.4838 - accuracy: 0.4762
1412/1563 [==========================>...] - ETA: 0s - loss: 1.4824 - accuracy: 0.4769
1452/1563 [==========================>...] - ETA: 0s - loss: 1.4826 - accuracy: 0.4763
1492/1563 [===========================>..] - ETA: 0s - loss: 1.4832 - accuracy: 0.4763
1532/1563 [============================>.] - ETA: 0s - loss: 1.4825 - accuracy: 0.4763
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4816 - accuracy: 0.4764
Epoch 5/10
1/1563 [..............................] - ETA: 1s - loss: 1.6585 - accuracy: 0.3125
41/1563 [..............................] - ETA: 1s - loss: 1.4441 - accuracy: 0.5000
81/1563 [>.............................] - ETA: 1s - loss: 1.4206 - accuracy: 0.5039
121/1563 [=>............................] - ETA: 1s - loss: 1.4242 - accuracy: 0.4972
162/1563 [==>...........................] - ETA: 1s - loss: 1.4310 - accuracy: 0.4931
202/1563 [==>...........................] - ETA: 1s - loss: 1.4387 - accuracy: 0.4910
242/1563 [===>..........................] - ETA: 1s - loss: 1.4445 - accuracy: 0.4919
282/1563 [====>.........................] - ETA: 1s - loss: 1.4419 - accuracy: 0.4955
322/1563 [=====>........................] - ETA: 1s - loss: 1.4413 - accuracy: 0.4928
361/1563 [=====>........................] - ETA: 1s - loss: 1.4432 - accuracy: 0.4932
400/1563 [======>.......................] - ETA: 1s - loss: 1.4426 - accuracy: 0.4937
439/1563 [=======>......................] - ETA: 1s - loss: 1.4425 - accuracy: 0.4937
478/1563 [========>.....................] - ETA: 1s - loss: 1.4432 - accuracy: 0.4927
518/1563 [========>.....................] - ETA: 1s - loss: 1.4434 - accuracy: 0.4928
557/1563 [=========>....................] - ETA: 1s - loss: 1.4437 - accuracy: 0.4924
596/1563 [==========>...................] - ETA: 1s - loss: 1.4455 - accuracy: 0.4924
635/1563 [===========>..................] - ETA: 1s - loss: 1.4472 - accuracy: 0.4932
675/1563 [===========>..................] - ETA: 1s - loss: 1.4474 - accuracy: 0.4924
715/1563 [============>.................] - ETA: 1s - loss: 1.4470 - accuracy: 0.4920
755/1563 [=============>................] - ETA: 1s - loss: 1.4466 - accuracy: 0.4920
794/1563 [==============>...............] - ETA: 0s - loss: 1.4452 - accuracy: 0.4927
833/1563 [==============>...............] - ETA: 0s - loss: 1.4439 - accuracy: 0.4928
872/1563 [===============>..............] - ETA: 0s - loss: 1.4452 - accuracy: 0.4916
911/1563 [================>.............] - ETA: 0s - loss: 1.4439 - accuracy: 0.4926
951/1563 [=================>............] - ETA: 0s - loss: 1.4431 - accuracy: 0.4926
990/1563 [==================>...........] - ETA: 0s - loss: 1.4424 - accuracy: 0.4934
1029/1563 [==================>...........] - ETA: 0s - loss: 1.4424 - accuracy: 0.4937
1068/1563 [===================>..........] - ETA: 0s - loss: 1.4413 - accuracy: 0.4933
1107/1563 [====================>.........] - ETA: 0s - loss: 1.4419 - accuracy: 0.4931
1146/1563 [====================>.........] - ETA: 0s - loss: 1.4405 - accuracy: 0.4936
1186/1563 [=====================>........] - ETA: 0s - loss: 1.4405 - accuracy: 0.4936
1226/1563 [======================>.......] - ETA: 0s - loss: 1.4375 - accuracy: 0.4938
1265/1563 [=======================>......] - ETA: 0s - loss: 1.4366 - accuracy: 0.4944
1304/1563 [========================>.....] - ETA: 0s - loss: 1.4374 - accuracy: 0.4943
1343/1563 [========================>.....] - ETA: 0s - loss: 1.4372 - accuracy: 0.4945
1383/1563 [=========================>....] - ETA: 0s - loss: 1.4371 - accuracy: 0.4946
1422/1563 [==========================>...] - ETA: 0s - loss: 1.4362 - accuracy: 0.4947
1461/1563 [===========================>..] - ETA: 0s - loss: 1.4354 - accuracy: 0.4947
1500/1563 [===========================>..] - ETA: 0s - loss: 1.4355 - accuracy: 0.4945
1540/1563 [============================>.] - ETA: 0s - loss: 1.4342 - accuracy: 0.4950
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4336 - accuracy: 0.4954
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.3733 - accuracy: 0.5312
40/1563 [..............................] - ETA: 1s - loss: 1.4504 - accuracy: 0.4969
79/1563 [>.............................] - ETA: 1s - loss: 1.4157 - accuracy: 0.5095
119/1563 [=>............................] - ETA: 1s - loss: 1.4209 - accuracy: 0.5055
158/1563 [==>...........................] - ETA: 1s - loss: 1.4255 - accuracy: 0.5061
197/1563 [==>...........................] - ETA: 1s - loss: 1.4168 - accuracy: 0.5119
236/1563 [===>..........................] - ETA: 1s - loss: 1.4142 - accuracy: 0.5093
275/1563 [====>.........................] - ETA: 1s - loss: 1.4046 - accuracy: 0.5120
314/1563 [=====>........................] - ETA: 1s - loss: 1.3990 - accuracy: 0.5142
354/1563 [=====>........................] - ETA: 1s - loss: 1.4007 - accuracy: 0.5132
393/1563 [======>.......................] - ETA: 1s - loss: 1.4001 - accuracy: 0.5115
432/1563 [=======>......................] - ETA: 1s - loss: 1.3984 - accuracy: 0.5116
471/1563 [========>.....................] - ETA: 1s - loss: 1.3959 - accuracy: 0.5117
510/1563 [========>.....................] - ETA: 1s - loss: 1.3937 - accuracy: 0.5121
549/1563 [=========>....................] - ETA: 1s - loss: 1.3913 - accuracy: 0.5134
588/1563 [==========>...................] - ETA: 1s - loss: 1.3929 - accuracy: 0.5129
627/1563 [===========>..................] - ETA: 1s - loss: 1.3955 - accuracy: 0.5118
668/1563 [===========>..................] - ETA: 1s - loss: 1.3960 - accuracy: 0.5111
707/1563 [============>.................] - ETA: 1s - loss: 1.3959 - accuracy: 0.5109
747/1563 [=============>................] - ETA: 1s - loss: 1.3970 - accuracy: 0.5105
788/1563 [==============>...............] - ETA: 0s - loss: 1.3958 - accuracy: 0.5106
829/1563 [==============>...............] - ETA: 0s - loss: 1.3962 - accuracy: 0.5097
870/1563 [===============>..............] - ETA: 0s - loss: 1.3964 - accuracy: 0.5100
910/1563 [================>.............] - ETA: 0s - loss: 1.3953 - accuracy: 0.5111
950/1563 [=================>............] - ETA: 0s - loss: 1.3940 - accuracy: 0.5114
990/1563 [==================>...........] - ETA: 0s - loss: 1.3957 - accuracy: 0.5109
1030/1563 [==================>...........] - ETA: 0s - loss: 1.3929 - accuracy: 0.5112
1070/1563 [===================>..........] - ETA: 0s - loss: 1.3941 - accuracy: 0.5103
1111/1563 [====================>.........] - ETA: 0s - loss: 1.3928 - accuracy: 0.5115
1152/1563 [=====================>........] - ETA: 0s - loss: 1.3937 - accuracy: 0.5111
1193/1563 [=====================>........] - ETA: 0s - loss: 1.3935 - accuracy: 0.5111
1233/1563 [======================>.......] - ETA: 0s - loss: 1.3920 - accuracy: 0.5117
1273/1563 [=======================>......] - ETA: 0s - loss: 1.3918 - accuracy: 0.5119
1313/1563 [========================>.....] - ETA: 0s - loss: 1.3919 - accuracy: 0.5120
1354/1563 [========================>.....] - ETA: 0s - loss: 1.3928 - accuracy: 0.5116
1394/1563 [=========================>....] - ETA: 0s - loss: 1.3934 - accuracy: 0.5108
1434/1563 [==========================>...] - ETA: 0s - loss: 1.3929 - accuracy: 0.5103
1474/1563 [===========================>..] - ETA: 0s - loss: 1.3915 - accuracy: 0.5109
1514/1563 [============================>.] - ETA: 0s - loss: 1.3919 - accuracy: 0.5105
1554/1563 [============================>.] - ETA: 0s - loss: 1.3904 - accuracy: 0.5113
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3901 - accuracy: 0.5114
Epoch 7/10
1/1563 [..............................] - ETA: 1s - loss: 1.0698 - accuracy: 0.5625
41/1563 [..............................] - ETA: 1s - loss: 1.3475 - accuracy: 0.5236
81/1563 [>.............................] - ETA: 1s - loss: 1.3599 - accuracy: 0.5193
121/1563 [=>............................] - ETA: 1s - loss: 1.3611 - accuracy: 0.5145
161/1563 [==>...........................] - ETA: 1s - loss: 1.3555 - accuracy: 0.5159
201/1563 [==>...........................] - ETA: 1s - loss: 1.3579 - accuracy: 0.5173
241/1563 [===>..........................] - ETA: 1s - loss: 1.3595 - accuracy: 0.5205
281/1563 [====>.........................] - ETA: 1s - loss: 1.3550 - accuracy: 0.5202
321/1563 [=====>........................] - ETA: 1s - loss: 1.3581 - accuracy: 0.5193
361/1563 [=====>........................] - ETA: 1s - loss: 1.3565 - accuracy: 0.5195
401/1563 [======>.......................] - ETA: 1s - loss: 1.3593 - accuracy: 0.5207
441/1563 [=======>......................] - ETA: 1s - loss: 1.3606 - accuracy: 0.5203
480/1563 [========>.....................] - ETA: 1s - loss: 1.3601 - accuracy: 0.5184
520/1563 [========>.....................] - ETA: 1s - loss: 1.3602 - accuracy: 0.5188
560/1563 [=========>....................] - ETA: 1s - loss: 1.3598 - accuracy: 0.5186
600/1563 [==========>...................] - ETA: 1s - loss: 1.3567 - accuracy: 0.5206
640/1563 [===========>..................] - ETA: 1s - loss: 1.3570 - accuracy: 0.5209
680/1563 [============>.................] - ETA: 1s - loss: 1.3566 - accuracy: 0.5211
721/1563 [============>.................] - ETA: 1s - loss: 1.3574 - accuracy: 0.5220
761/1563 [=============>................] - ETA: 1s - loss: 1.3550 - accuracy: 0.5233
801/1563 [==============>...............] - ETA: 0s - loss: 1.3541 - accuracy: 0.5234
841/1563 [===============>..............] - ETA: 0s - loss: 1.3552 - accuracy: 0.5223
881/1563 [===============>..............] - ETA: 0s - loss: 1.3552 - accuracy: 0.5231
921/1563 [================>.............] - ETA: 0s - loss: 1.3556 - accuracy: 0.5223
960/1563 [=================>............] - ETA: 0s - loss: 1.3541 - accuracy: 0.5229
1000/1563 [==================>...........] - ETA: 0s - loss: 1.3525 - accuracy: 0.5234
1040/1563 [==================>...........] - ETA: 0s - loss: 1.3515 - accuracy: 0.5235
1080/1563 [===================>..........] - ETA: 0s - loss: 1.3517 - accuracy: 0.5239
1120/1563 [====================>.........] - ETA: 0s - loss: 1.3528 - accuracy: 0.5238
1160/1563 [=====================>........] - ETA: 0s - loss: 1.3533 - accuracy: 0.5241
1200/1563 [======================>.......] - ETA: 0s - loss: 1.3522 - accuracy: 0.5251
1240/1563 [======================>.......] - ETA: 0s - loss: 1.3522 - accuracy: 0.5251
1280/1563 [=======================>......] - ETA: 0s - loss: 1.3529 - accuracy: 0.5248
1320/1563 [========================>.....] - ETA: 0s - loss: 1.3525 - accuracy: 0.5251
1360/1563 [=========================>....] - ETA: 0s - loss: 1.3522 - accuracy: 0.5247
1399/1563 [=========================>....] - ETA: 0s - loss: 1.3515 - accuracy: 0.5246
1439/1563 [==========================>...] - ETA: 0s - loss: 1.3514 - accuracy: 0.5249
1479/1563 [===========================>..] - ETA: 0s - loss: 1.3501 - accuracy: 0.5255
1519/1563 [============================>.] - ETA: 0s - loss: 1.3507 - accuracy: 0.5257
1558/1563 [============================>.] - ETA: 0s - loss: 1.3512 - accuracy: 0.5252
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3513 - accuracy: 0.5252
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.6233 - accuracy: 0.4688
40/1563 [..............................] - ETA: 1s - loss: 1.3084 - accuracy: 0.5414
79/1563 [>.............................] - ETA: 1s - loss: 1.3166 - accuracy: 0.5332
118/1563 [=>............................] - ETA: 1s - loss: 1.3179 - accuracy: 0.5368
157/1563 [==>...........................] - ETA: 1s - loss: 1.3241 - accuracy: 0.5334
196/1563 [==>...........................] - ETA: 1s - loss: 1.3219 - accuracy: 0.5362
235/1563 [===>..........................] - ETA: 1s - loss: 1.3184 - accuracy: 0.5383
274/1563 [====>.........................] - ETA: 1s - loss: 1.3194 - accuracy: 0.5384
313/1563 [=====>........................] - ETA: 1s - loss: 1.3241 - accuracy: 0.5354
352/1563 [=====>........................] - ETA: 1s - loss: 1.3261 - accuracy: 0.5336
391/1563 [======>.......................] - ETA: 1s - loss: 1.3200 - accuracy: 0.5365
430/1563 [=======>......................] - ETA: 1s - loss: 1.3225 - accuracy: 0.5360
469/1563 [========>.....................] - ETA: 1s - loss: 1.3208 - accuracy: 0.5366
507/1563 [========>.....................] - ETA: 1s - loss: 1.3221 - accuracy: 0.5359
546/1563 [=========>....................] - ETA: 1s - loss: 1.3227 - accuracy: 0.5367
585/1563 [==========>...................] - ETA: 1s - loss: 1.3215 - accuracy: 0.5372
625/1563 [==========>...................] - ETA: 1s - loss: 1.3214 - accuracy: 0.5385
665/1563 [===========>..................] - ETA: 1s - loss: 1.3214 - accuracy: 0.5383
705/1563 [============>.................] - ETA: 1s - loss: 1.3223 - accuracy: 0.5372
745/1563 [=============>................] - ETA: 1s - loss: 1.3228 - accuracy: 0.5367
785/1563 [==============>...............] - ETA: 1s - loss: 1.3207 - accuracy: 0.5373
825/1563 [==============>...............] - ETA: 0s - loss: 1.3205 - accuracy: 0.5374
865/1563 [===============>..............] - ETA: 0s - loss: 1.3209 - accuracy: 0.5372
905/1563 [================>.............] - ETA: 0s - loss: 1.3220 - accuracy: 0.5368
945/1563 [=================>............] - ETA: 0s - loss: 1.3234 - accuracy: 0.5365
984/1563 [=================>............] - ETA: 0s - loss: 1.3232 - accuracy: 0.5359
1024/1563 [==================>...........] - ETA: 0s - loss: 1.3239 - accuracy: 0.5354
1064/1563 [===================>..........] - ETA: 0s - loss: 1.3249 - accuracy: 0.5357
1104/1563 [====================>.........] - ETA: 0s - loss: 1.3264 - accuracy: 0.5350
1144/1563 [====================>.........] - ETA: 0s - loss: 1.3258 - accuracy: 0.5355
1184/1563 [=====================>........] - ETA: 0s - loss: 1.3260 - accuracy: 0.5351
1224/1563 [======================>.......] - ETA: 0s - loss: 1.3241 - accuracy: 0.5354
1264/1563 [=======================>......] - ETA: 0s - loss: 1.3219 - accuracy: 0.5358
1304/1563 [========================>.....] - ETA: 0s - loss: 1.3205 - accuracy: 0.5365
1344/1563 [========================>.....] - ETA: 0s - loss: 1.3199 - accuracy: 0.5368
1384/1563 [=========================>....] - ETA: 0s - loss: 1.3174 - accuracy: 0.5373
1424/1563 [==========================>...] - ETA: 0s - loss: 1.3172 - accuracy: 0.5374
1464/1563 [===========================>..] - ETA: 0s - loss: 1.3172 - accuracy: 0.5373
1503/1563 [===========================>..] - ETA: 0s - loss: 1.3177 - accuracy: 0.5373
1542/1563 [============================>.] - ETA: 0s - loss: 1.3174 - accuracy: 0.5373
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3169 - accuracy: 0.5376
Epoch 9/10
1/1563 [..............................] - ETA: 2s - loss: 1.2670 - accuracy: 0.4062
40/1563 [..............................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5625
78/1563 [>.............................] - ETA: 1s - loss: 1.2670 - accuracy: 0.5613
117/1563 [=>............................] - ETA: 1s - loss: 1.2733 - accuracy: 0.5553
156/1563 [=>............................] - ETA: 1s - loss: 1.2701 - accuracy: 0.5539
194/1563 [==>...........................] - ETA: 1s - loss: 1.2756 - accuracy: 0.5557
233/1563 [===>..........................] - ETA: 1s - loss: 1.2794 - accuracy: 0.5541
272/1563 [====>.........................] - ETA: 1s - loss: 1.2808 - accuracy: 0.5546
311/1563 [====>.........................] - ETA: 1s - loss: 1.2800 - accuracy: 0.5550
349/1563 [=====>........................] - ETA: 1s - loss: 1.2817 - accuracy: 0.5539
388/1563 [======>.......................] - ETA: 1s - loss: 1.2884 - accuracy: 0.5500
427/1563 [=======>......................] - ETA: 1s - loss: 1.2917 - accuracy: 0.5485
466/1563 [=======>......................] - ETA: 1s - loss: 1.2898 - accuracy: 0.5484
505/1563 [========>.....................] - ETA: 1s - loss: 1.2888 - accuracy: 0.5475
544/1563 [=========>....................] - ETA: 1s - loss: 1.2867 - accuracy: 0.5490
583/1563 [==========>...................] - ETA: 1s - loss: 1.2846 - accuracy: 0.5500
622/1563 [==========>...................] - ETA: 1s - loss: 1.2830 - accuracy: 0.5509
661/1563 [===========>..................] - ETA: 1s - loss: 1.2820 - accuracy: 0.5517
700/1563 [============>.................] - ETA: 1s - loss: 1.2822 - accuracy: 0.5510
739/1563 [=============>................] - ETA: 1s - loss: 1.2842 - accuracy: 0.5493
778/1563 [=============>................] - ETA: 1s - loss: 1.2860 - accuracy: 0.5483
817/1563 [==============>...............] - ETA: 0s - loss: 1.2839 - accuracy: 0.5489
856/1563 [===============>..............] - ETA: 0s - loss: 1.2866 - accuracy: 0.5478
895/1563 [================>.............] - ETA: 0s - loss: 1.2851 - accuracy: 0.5473
934/1563 [================>.............] - ETA: 0s - loss: 1.2861 - accuracy: 0.5474
973/1563 [=================>............] - ETA: 0s - loss: 1.2862 - accuracy: 0.5471
1012/1563 [==================>...........] - ETA: 0s - loss: 1.2851 - accuracy: 0.5469
1050/1563 [===================>..........] - ETA: 0s - loss: 1.2838 - accuracy: 0.5478
1089/1563 [===================>..........] - ETA: 0s - loss: 1.2840 - accuracy: 0.5482
1128/1563 [====================>.........] - ETA: 0s - loss: 1.2843 - accuracy: 0.5481
1167/1563 [=====================>........] - ETA: 0s - loss: 1.2846 - accuracy: 0.5479
1206/1563 [======================>.......] - ETA: 0s - loss: 1.2833 - accuracy: 0.5486
1245/1563 [======================>.......] - ETA: 0s - loss: 1.2829 - accuracy: 0.5488
1284/1563 [=======================>......] - ETA: 0s - loss: 1.2819 - accuracy: 0.5491
1323/1563 [========================>.....] - ETA: 0s - loss: 1.2827 - accuracy: 0.5488
1361/1563 [=========================>....] - ETA: 0s - loss: 1.2826 - accuracy: 0.5493
1399/1563 [=========================>....] - ETA: 0s - loss: 1.2837 - accuracy: 0.5490
1436/1563 [==========================>...] - ETA: 0s - loss: 1.2833 - accuracy: 0.5491
1474/1563 [===========================>..] - ETA: 0s - loss: 1.2841 - accuracy: 0.5481
1513/1563 [============================>.] - ETA: 0s - loss: 1.2841 - accuracy: 0.5481
1552/1563 [============================>.] - ETA: 0s - loss: 1.2841 - accuracy: 0.5480
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2836 - accuracy: 0.5481
Epoch 10/10
1/1563 [..............................] - ETA: 2s - loss: 1.5916 - accuracy: 0.5000
40/1563 [..............................] - ETA: 1s - loss: 1.2121 - accuracy: 0.5758
79/1563 [>.............................] - ETA: 1s - loss: 1.2103 - accuracy: 0.5720
118/1563 [=>............................] - ETA: 1s - loss: 1.2104 - accuracy: 0.5720
157/1563 [==>...........................] - ETA: 1s - loss: 1.2222 - accuracy: 0.5719
196/1563 [==>...........................] - ETA: 1s - loss: 1.2269 - accuracy: 0.5679
235/1563 [===>..........................] - ETA: 1s - loss: 1.2388 - accuracy: 0.5638
273/1563 [====>.........................] - ETA: 1s - loss: 1.2415 - accuracy: 0.5618
312/1563 [====>.........................] - ETA: 1s - loss: 1.2396 - accuracy: 0.5599
351/1563 [=====>........................] - ETA: 1s - loss: 1.2444 - accuracy: 0.5597
390/1563 [======>.......................] - ETA: 1s - loss: 1.2460 - accuracy: 0.5615
429/1563 [=======>......................] - ETA: 1s - loss: 1.2479 - accuracy: 0.5610
468/1563 [=======>......................] - ETA: 1s - loss: 1.2492 - accuracy: 0.5607
507/1563 [========>.....................] - ETA: 1s - loss: 1.2494 - accuracy: 0.5615
546/1563 [=========>....................] - ETA: 1s - loss: 1.2512 - accuracy: 0.5610
585/1563 [==========>...................] - ETA: 1s - loss: 1.2505 - accuracy: 0.5623
624/1563 [==========>...................] - ETA: 1s - loss: 1.2508 - accuracy: 0.5623
663/1563 [===========>..................] - ETA: 1s - loss: 1.2551 - accuracy: 0.5623
702/1563 [============>.................] - ETA: 1s - loss: 1.2536 - accuracy: 0.5625
741/1563 [=============>................] - ETA: 1s - loss: 1.2511 - accuracy: 0.5636
780/1563 [=============>................] - ETA: 1s - loss: 1.2529 - accuracy: 0.5629
819/1563 [==============>...............] - ETA: 0s - loss: 1.2536 - accuracy: 0.5634
858/1563 [===============>..............] - ETA: 0s - loss: 1.2510 - accuracy: 0.5643
897/1563 [================>.............] - ETA: 0s - loss: 1.2513 - accuracy: 0.5640
935/1563 [================>.............] - ETA: 0s - loss: 1.2520 - accuracy: 0.5638
974/1563 [=================>............] - ETA: 0s - loss: 1.2515 - accuracy: 0.5637
1013/1563 [==================>...........] - ETA: 0s - loss: 1.2509 - accuracy: 0.5636
1052/1563 [===================>..........] - ETA: 0s - loss: 1.2513 - accuracy: 0.5637
1091/1563 [===================>..........] - ETA: 0s - loss: 1.2534 - accuracy: 0.5626
1130/1563 [====================>.........] - ETA: 0s - loss: 1.2506 - accuracy: 0.5634
1169/1563 [=====================>........] - ETA: 0s - loss: 1.2517 - accuracy: 0.5621
1208/1563 [======================>.......] - ETA: 0s - loss: 1.2528 - accuracy: 0.5616
1247/1563 [======================>.......] - ETA: 0s - loss: 1.2535 - accuracy: 0.5615
1286/1563 [=======================>......] - ETA: 0s - loss: 1.2512 - accuracy: 0.5626
1325/1563 [========================>.....] - ETA: 0s - loss: 1.2507 - accuracy: 0.5625
1365/1563 [=========================>....] - ETA: 0s - loss: 1.2506 - accuracy: 0.5627
1405/1563 [=========================>....] - ETA: 0s - loss: 1.2499 - accuracy: 0.5636
1445/1563 [==========================>...] - ETA: 0s - loss: 1.2502 - accuracy: 0.5631
1485/1563 [===========================>..] - ETA: 0s - loss: 1.2510 - accuracy: 0.5631
1525/1563 [============================>.] - ETA: 0s - loss: 1.2515 - accuracy: 0.5630
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2507 - accuracy: 0.5634
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 36.206727027893066
hydro04
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins4276250844956316637.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
