Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 97683 queued and waiting for resources srun: job 97683 has been allocated resources 2024-11-16 07:43:04.657149: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-16 07:43:05.878319: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-16 07:43:08.704501: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-16 07:43:08.705828: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-16 07:43:08.705836: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-16 07:43:15.085048: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-16 07:43:16.438269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-16 07:43:21.175015: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:33:50 - loss: 2.3078 - accuracy: 0.0625 37/1563 [..............................] - ETA: 2s - loss: 2.2384 - accuracy: 0.1782 74/1563 [>.............................] - ETA: 2s - loss: 2.1754 - accuracy: 0.2078 111/1563 [=>............................] - ETA: 2s - loss: 2.1205 - accuracy: 0.2252 148/1563 [=>............................] - ETA: 1s - loss: 2.0986 - accuracy: 0.2369 185/1563 [==>...........................] - ETA: 1s - loss: 2.0733 - accuracy: 0.2473 222/1563 [===>..........................] - ETA: 1s - loss: 2.0511 - accuracy: 0.2568 259/1563 [===>..........................] - ETA: 1s - loss: 2.0308 - accuracy: 0.2627 296/1563 [====>.........................] - ETA: 1s - loss: 2.0136 - accuracy: 0.2719 333/1563 [=====>........................] - ETA: 1s - loss: 1.9976 - accuracy: 0.2784 370/1563 [======>.......................] - ETA: 1s - loss: 1.9862 - accuracy: 0.2839 407/1563 [======>.......................] - ETA: 1s - loss: 1.9787 - accuracy: 0.2888 444/1563 [=======>......................] - ETA: 1s - loss: 1.9691 - accuracy: 0.2919 481/1563 [========>.....................] - ETA: 1s - loss: 1.9598 - accuracy: 0.2950 518/1563 [========>.....................] - ETA: 1s - loss: 1.9500 - accuracy: 0.2994 555/1563 [=========>....................] - ETA: 1s - loss: 1.9406 - accuracy: 0.3026 592/1563 [==========>...................] - ETA: 1s - loss: 1.9339 - accuracy: 0.3047 629/1563 [===========>..................] - ETA: 1s - loss: 1.9266 - accuracy: 0.3087 666/1563 [===========>..................] - ETA: 1s - loss: 1.9212 - accuracy: 0.3109 703/1563 [============>.................] - ETA: 1s - loss: 1.9143 - accuracy: 0.3134 741/1563 [=============>................] - ETA: 1s - loss: 1.9074 - accuracy: 0.3165 778/1563 [=============>................] - ETA: 1s - loss: 1.8987 - accuracy: 0.3198 815/1563 [==============>...............] - ETA: 1s - loss: 1.8939 - accuracy: 0.3224 852/1563 [===============>..............] - ETA: 0s - loss: 1.8901 - accuracy: 0.3238 889/1563 [================>.............] - ETA: 0s - loss: 1.8845 - accuracy: 0.3254 926/1563 [================>.............] - ETA: 0s - loss: 1.8794 - accuracy: 0.3272 964/1563 [=================>............] - ETA: 0s - loss: 1.8744 - accuracy: 0.3290 1001/1563 [==================>...........] - ETA: 0s - loss: 1.8711 - accuracy: 0.3306 1038/1563 [==================>...........] - ETA: 0s - loss: 1.8666 - accuracy: 0.3326 1075/1563 [===================>..........] - ETA: 0s - loss: 1.8625 - accuracy: 0.3337 1112/1563 [====================>.........] - ETA: 0s - loss: 1.8577 - accuracy: 0.3353 1150/1563 [=====================>........] - ETA: 0s - loss: 1.8533 - accuracy: 0.3370 1187/1563 [=====================>........] - ETA: 0s - loss: 1.8493 - accuracy: 0.3391 1225/1563 [======================>.......] - ETA: 0s - loss: 1.8454 - accuracy: 0.3415 1262/1563 [=======================>......] - ETA: 0s - loss: 1.8416 - accuracy: 0.3434 1300/1563 [=======================>......] - ETA: 0s - loss: 1.8386 - accuracy: 0.3444 1337/1563 [========================>.....] - ETA: 0s - loss: 1.8351 - accuracy: 0.3455 1374/1563 [=========================>....] - ETA: 0s - loss: 1.8322 - accuracy: 0.3466 1411/1563 [==========================>...] - ETA: 0s - loss: 1.8300 - accuracy: 0.3476 1448/1563 [==========================>...] - ETA: 0s - loss: 1.8269 - accuracy: 0.3490 1486/1563 [===========================>..] - ETA: 0s - loss: 1.8241 - accuracy: 0.3505 1524/1563 [============================>.] - ETA: 0s - loss: 1.8198 - accuracy: 0.3518 1561/1563 [============================>.] - ETA: 0s - loss: 1.8160 - accuracy: 0.3528 1563/1563 [==============================] - 6s 1ms/step - loss: 1.8157 - accuracy: 0.3528 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.9127 - accuracy: 0.2812 38/1563 [..............................] - ETA: 2s - loss: 1.6918 - accuracy: 0.4104 76/1563 [>.............................] - ETA: 2s - loss: 1.6809 - accuracy: 0.4100 114/1563 [=>............................] - ETA: 1s - loss: 1.6649 - accuracy: 0.4164 152/1563 [=>............................] - ETA: 1s - loss: 1.6733 - accuracy: 0.4089 190/1563 [==>...........................] - ETA: 1s - loss: 1.6723 - accuracy: 0.4090 227/1563 [===>..........................] - ETA: 1s - loss: 1.6689 - accuracy: 0.4115 264/1563 [====>.........................] - ETA: 1s - loss: 1.6670 - accuracy: 0.4111 301/1563 [====>.........................] - ETA: 1s - loss: 1.6647 - accuracy: 0.4119 338/1563 [=====>........................] - ETA: 1s - loss: 1.6620 - accuracy: 0.4115 375/1563 [======>.......................] - ETA: 1s - loss: 1.6630 - accuracy: 0.4120 413/1563 [======>.......................] - ETA: 1s - loss: 1.6557 - accuracy: 0.4153 450/1563 [=======>......................] - ETA: 1s - loss: 1.6547 - accuracy: 0.4172 487/1563 [========>.....................] - ETA: 1s - loss: 1.6515 - accuracy: 0.4185 524/1563 [=========>....................] - ETA: 1s - loss: 1.6484 - accuracy: 0.4200 561/1563 [=========>....................] - ETA: 1s - loss: 1.6503 - accuracy: 0.4182 598/1563 [==========>...................] - ETA: 1s - loss: 1.6486 - accuracy: 0.4185 635/1563 [===========>..................] - ETA: 1s - loss: 1.6495 - accuracy: 0.4175 672/1563 [===========>..................] - ETA: 1s - loss: 1.6483 - accuracy: 0.4178 709/1563 [============>.................] - ETA: 1s - loss: 1.6482 - accuracy: 0.4183 746/1563 [=============>................] - ETA: 1s - loss: 1.6487 - accuracy: 0.4185 783/1563 [==============>...............] - ETA: 1s - loss: 1.6492 - accuracy: 0.4189 820/1563 [==============>...............] - ETA: 1s - loss: 1.6459 - accuracy: 0.4198 857/1563 [===============>..............] - ETA: 0s - loss: 1.6439 - accuracy: 0.4200 894/1563 [================>.............] - ETA: 0s - loss: 1.6431 - accuracy: 0.4196 931/1563 [================>.............] - ETA: 0s - loss: 1.6429 - accuracy: 0.4192 968/1563 [=================>............] - ETA: 0s - loss: 1.6421 - accuracy: 0.4200 1005/1563 [==================>...........] - ETA: 0s - loss: 1.6393 - accuracy: 0.4209 1042/1563 [===================>..........] - ETA: 0s - loss: 1.6381 - accuracy: 0.4215 1079/1563 [===================>..........] - ETA: 0s - loss: 1.6352 - accuracy: 0.4227 1116/1563 [====================>.........] - ETA: 0s - loss: 1.6336 - accuracy: 0.4237 1153/1563 [=====================>........] - ETA: 0s - loss: 1.6312 - accuracy: 0.4246 1191/1563 [=====================>........] - ETA: 0s - loss: 1.6299 - accuracy: 0.4255 1228/1563 [======================>.......] - ETA: 0s - loss: 1.6280 - accuracy: 0.4263 1265/1563 [=======================>......] - ETA: 0s - loss: 1.6284 - accuracy: 0.4262 1302/1563 [=======================>......] - ETA: 0s - loss: 1.6282 - accuracy: 0.4260 1339/1563 [========================>.....] - ETA: 0s - loss: 1.6284 - accuracy: 0.4257 1376/1563 [=========================>....] - ETA: 0s - loss: 1.6271 - accuracy: 0.4258 1413/1563 [==========================>...] - ETA: 0s - loss: 1.6274 - accuracy: 0.4260 1450/1563 [==========================>...] - ETA: 0s - loss: 1.6262 - accuracy: 0.4263 1487/1563 [===========================>..] - ETA: 0s - loss: 1.6260 - accuracy: 0.4263 1524/1563 [============================>.] - ETA: 0s - loss: 1.6254 - accuracy: 0.4265 1561/1563 [============================>.] - ETA: 0s - loss: 1.6246 - accuracy: 0.4269 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6244 - accuracy: 0.4270 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.5625 - accuracy: 0.4375 38/1563 [..............................] - ETA: 2s - loss: 1.5583 - accuracy: 0.4391 76/1563 [>.............................] - ETA: 2s - loss: 1.5652 - accuracy: 0.4346 114/1563 [=>............................] - ETA: 1s - loss: 1.5697 - accuracy: 0.4391 152/1563 [=>............................] - ETA: 1s - loss: 1.5579 - accuracy: 0.4449 190/1563 [==>...........................] - ETA: 1s - loss: 1.5605 - accuracy: 0.4469 228/1563 [===>..........................] - ETA: 1s - loss: 1.5593 - accuracy: 0.4456 266/1563 [====>.........................] - ETA: 1s - loss: 1.5563 - accuracy: 0.4482 304/1563 [====>.........................] - ETA: 1s - loss: 1.5543 - accuracy: 0.4484 342/1563 [=====>........................] - ETA: 1s - loss: 1.5490 - accuracy: 0.4518 379/1563 [======>.......................] - ETA: 1s - loss: 1.5495 - accuracy: 0.4507 416/1563 [======>.......................] - ETA: 1s - loss: 1.5500 - accuracy: 0.4519 453/1563 [=======>......................] - ETA: 1s - loss: 1.5467 - accuracy: 0.4537 490/1563 [========>.....................] - ETA: 1s - loss: 1.5457 - accuracy: 0.4544 527/1563 [=========>....................] - ETA: 1s - loss: 1.5461 - accuracy: 0.4533 565/1563 [=========>....................] - ETA: 1s - loss: 1.5443 - accuracy: 0.4531 602/1563 [==========>...................] - ETA: 1s - loss: 1.5434 - accuracy: 0.4534 639/1563 [===========>..................] - ETA: 1s - loss: 1.5433 - accuracy: 0.4527 676/1563 [===========>..................] - ETA: 1s - loss: 1.5425 - accuracy: 0.4532 713/1563 [============>.................] - ETA: 1s - loss: 1.5450 - accuracy: 0.4529 750/1563 [=============>................] - ETA: 1s - loss: 1.5463 - accuracy: 0.4526 787/1563 [==============>...............] - ETA: 1s - loss: 1.5471 - accuracy: 0.4529 824/1563 [==============>...............] - ETA: 1s - loss: 1.5464 - accuracy: 0.4530 862/1563 [===============>..............] - ETA: 0s - loss: 1.5477 - accuracy: 0.4522 900/1563 [================>.............] - ETA: 0s - loss: 1.5474 - accuracy: 0.4517 937/1563 [================>.............] - ETA: 0s - loss: 1.5467 - accuracy: 0.4523 974/1563 [=================>............] - ETA: 0s - loss: 1.5455 - accuracy: 0.4525 1011/1563 [==================>...........] - ETA: 0s - loss: 1.5452 - accuracy: 0.4527 1048/1563 [===================>..........] - ETA: 0s - loss: 1.5454 - accuracy: 0.4532 1085/1563 [===================>..........] - ETA: 0s - loss: 1.5450 - accuracy: 0.4536 1122/1563 [====================>.........] - ETA: 0s - loss: 1.5452 - accuracy: 0.4538 1159/1563 [=====================>........] - ETA: 0s - loss: 1.5454 - accuracy: 0.4536 1196/1563 [=====================>........] - ETA: 0s - loss: 1.5455 - accuracy: 0.4539 1233/1563 [======================>.......] - ETA: 0s - loss: 1.5456 - accuracy: 0.4543 1270/1563 [=======================>......] - ETA: 0s - loss: 1.5453 - accuracy: 0.4543 1307/1563 [========================>.....] - ETA: 0s - loss: 1.5437 - accuracy: 0.4552 1344/1563 [========================>.....] - ETA: 0s - loss: 1.5432 - accuracy: 0.4546 1381/1563 [=========================>....] - ETA: 0s - loss: 1.5421 - accuracy: 0.4553 1418/1563 [==========================>...] - ETA: 0s - loss: 1.5411 - accuracy: 0.4555 1455/1563 [==========================>...] - ETA: 0s - loss: 1.5412 - accuracy: 0.4553 1492/1563 [===========================>..] - ETA: 0s - loss: 1.5413 - accuracy: 0.4553 1529/1563 [============================>.] - ETA: 0s - loss: 1.5414 - accuracy: 0.4558 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5413 - accuracy: 0.4558 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.3503 - accuracy: 0.5625 38/1563 [..............................] - ETA: 2s - loss: 1.4436 - accuracy: 0.4942 75/1563 [>.............................] - ETA: 2s - loss: 1.4502 - accuracy: 0.4850 112/1563 [=>............................] - ETA: 1s - loss: 1.4576 - accuracy: 0.4891 149/1563 [=>............................] - ETA: 1s - loss: 1.4762 - accuracy: 0.4830 186/1563 [==>...........................] - ETA: 1s - loss: 1.4669 - accuracy: 0.4864 223/1563 [===>..........................] - ETA: 1s - loss: 1.4777 - accuracy: 0.4807 260/1563 [===>..........................] - ETA: 1s - loss: 1.4782 - accuracy: 0.4792 297/1563 [====>.........................] - ETA: 1s - loss: 1.4789 - accuracy: 0.4782 334/1563 [=====>........................] - ETA: 1s - loss: 1.4802 - accuracy: 0.4781 371/1563 [======>.......................] - ETA: 1s - loss: 1.4821 - accuracy: 0.4770 408/1563 [======>.......................] - ETA: 1s - loss: 1.4868 - accuracy: 0.4760 445/1563 [=======>......................] - ETA: 1s - loss: 1.4834 - accuracy: 0.4758 483/1563 [========>.....................] - ETA: 1s - loss: 1.4862 - accuracy: 0.4741 521/1563 [=========>....................] - ETA: 1s - loss: 1.4885 - accuracy: 0.4741 558/1563 [=========>....................] - ETA: 1s - loss: 1.4863 - accuracy: 0.4746 595/1563 [==========>...................] - ETA: 1s - loss: 1.4865 - accuracy: 0.4748 632/1563 [===========>..................] - ETA: 1s - loss: 1.4894 - accuracy: 0.4744 669/1563 [===========>..................] - ETA: 1s - loss: 1.4882 - accuracy: 0.4742 706/1563 [============>.................] - ETA: 1s - loss: 1.4878 - accuracy: 0.4746 744/1563 [=============>................] - ETA: 1s - loss: 1.4857 - accuracy: 0.4752 781/1563 [=============>................] - ETA: 1s - loss: 1.4840 - accuracy: 0.4757 818/1563 [==============>...............] - ETA: 1s - loss: 1.4840 - accuracy: 0.4750 855/1563 [===============>..............] - ETA: 0s - loss: 1.4854 - accuracy: 0.4742 892/1563 [================>.............] - ETA: 0s - loss: 1.4872 - accuracy: 0.4746 929/1563 [================>.............] - ETA: 0s - loss: 1.4882 - accuracy: 0.4744 966/1563 [=================>............] - ETA: 0s - loss: 1.4883 - accuracy: 0.4743 1000/1563 [==================>...........] - ETA: 0s - loss: 1.4892 - accuracy: 0.4734 1037/1563 [==================>...........] - ETA: 0s - loss: 1.4889 - accuracy: 0.4738 1075/1563 [===================>..........] - ETA: 0s - loss: 1.4881 - accuracy: 0.4737 1112/1563 [====================>.........] - ETA: 0s - loss: 1.4881 - accuracy: 0.4733 1149/1563 [=====================>........] - ETA: 0s - loss: 1.4877 - accuracy: 0.4736 1186/1563 [=====================>........] - ETA: 0s - loss: 1.4880 - accuracy: 0.4734 1223/1563 [======================>.......] - ETA: 0s - loss: 1.4882 - accuracy: 0.4737 1260/1563 [=======================>......] - ETA: 0s - loss: 1.4864 - accuracy: 0.4748 1297/1563 [=======================>......] - ETA: 0s - loss: 1.4875 - accuracy: 0.4751 1334/1563 [========================>.....] - ETA: 0s - loss: 1.4868 - accuracy: 0.4754 1372/1563 [=========================>....] - ETA: 0s - loss: 1.4868 - accuracy: 0.4759 1409/1563 [==========================>...] - ETA: 0s - loss: 1.4840 - accuracy: 0.4773 1447/1563 [==========================>...] - ETA: 0s - loss: 1.4834 - accuracy: 0.4772 1484/1563 [===========================>..] - ETA: 0s - loss: 1.4839 - accuracy: 0.4772 1521/1563 [============================>.] - ETA: 0s - loss: 1.4835 - accuracy: 0.4769 1558/1563 [============================>.] - ETA: 0s - loss: 1.4834 - accuracy: 0.4774 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4832 - accuracy: 0.4774 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.5315 - accuracy: 0.5312 38/1563 [..............................] - ETA: 2s - loss: 1.4632 - accuracy: 0.4918 75/1563 [>.............................] - ETA: 2s - loss: 1.4449 - accuracy: 0.4975 112/1563 [=>............................] - ETA: 1s - loss: 1.4559 - accuracy: 0.4944 149/1563 [=>............................] - ETA: 1s - loss: 1.4700 - accuracy: 0.4870 187/1563 [==>...........................] - ETA: 1s - loss: 1.4577 - accuracy: 0.4928 224/1563 [===>..........................] - ETA: 1s - loss: 1.4598 - accuracy: 0.4923 261/1563 [====>.........................] - ETA: 1s - loss: 1.4659 - accuracy: 0.4899 298/1563 [====>.........................] - ETA: 1s - loss: 1.4648 - accuracy: 0.4891 335/1563 [=====>........................] - ETA: 1s - loss: 1.4623 - accuracy: 0.4875 372/1563 [======>.......................] - ETA: 1s - loss: 1.4547 - accuracy: 0.4887 408/1563 [======>.......................] - ETA: 1s - loss: 1.4562 - accuracy: 0.4882 446/1563 [=======>......................] - ETA: 1s - loss: 1.4579 - accuracy: 0.4886 484/1563 [========>.....................] - ETA: 1s - loss: 1.4593 - accuracy: 0.4875 522/1563 [=========>....................] - ETA: 1s - loss: 1.4544 - accuracy: 0.4893 560/1563 [=========>....................] - ETA: 1s - loss: 1.4507 - accuracy: 0.4898 597/1563 [==========>...................] - ETA: 1s - loss: 1.4507 - accuracy: 0.4901 635/1563 [===========>..................] - ETA: 1s - loss: 1.4504 - accuracy: 0.4903 673/1563 [===========>..................] - ETA: 1s - loss: 1.4517 - accuracy: 0.4910 711/1563 [============>.................] - ETA: 1s - loss: 1.4526 - accuracy: 0.4908 748/1563 [=============>................] - ETA: 1s - loss: 1.4530 - accuracy: 0.4912 786/1563 [==============>...............] - ETA: 1s - loss: 1.4553 - accuracy: 0.4906 824/1563 [==============>...............] - ETA: 0s - loss: 1.4549 - accuracy: 0.4897 862/1563 [===============>..............] - ETA: 0s - loss: 1.4542 - accuracy: 0.4895 900/1563 [================>.............] - ETA: 0s - loss: 1.4520 - accuracy: 0.4901 938/1563 [=================>............] - ETA: 0s - loss: 1.4497 - accuracy: 0.4903 976/1563 [=================>............] - ETA: 0s - loss: 1.4475 - accuracy: 0.4913 1014/1563 [==================>...........] - ETA: 0s - loss: 1.4468 - accuracy: 0.4921 1052/1563 [===================>..........] - ETA: 0s - loss: 1.4459 - accuracy: 0.4922 1090/1563 [===================>..........] - ETA: 0s - loss: 1.4443 - accuracy: 0.4926 1128/1563 [====================>.........] - ETA: 0s - loss: 1.4420 - accuracy: 0.4933 1166/1563 [=====================>........] - ETA: 0s - loss: 1.4411 - accuracy: 0.4933 1204/1563 [======================>.......] - ETA: 0s - loss: 1.4406 - accuracy: 0.4932 1242/1563 [======================>.......] - ETA: 0s - loss: 1.4401 - accuracy: 0.4936 1278/1563 [=======================>......] - ETA: 0s - loss: 1.4389 - accuracy: 0.4940 1315/1563 [========================>.....] - ETA: 0s - loss: 1.4393 - accuracy: 0.4942 1353/1563 [========================>.....] - ETA: 0s - loss: 1.4393 - accuracy: 0.4944 1391/1563 [=========================>....] - ETA: 0s - loss: 1.4393 - accuracy: 0.4939 1429/1563 [==========================>...] - ETA: 0s - loss: 1.4371 - accuracy: 0.4945 1467/1563 [===========================>..] - ETA: 0s - loss: 1.4362 - accuracy: 0.4946 1505/1563 [===========================>..] - ETA: 0s - loss: 1.4352 - accuracy: 0.4953 1543/1563 [============================>.] - ETA: 0s - loss: 1.4347 - accuracy: 0.4952 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4341 - accuracy: 0.4953 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.2854 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.3833 - accuracy: 0.5080 77/1563 [>.............................] - ETA: 1s - loss: 1.3893 - accuracy: 0.5114 115/1563 [=>............................] - ETA: 1s - loss: 1.3820 - accuracy: 0.5122 153/1563 [=>............................] - ETA: 1s - loss: 1.3912 - accuracy: 0.5112 191/1563 [==>...........................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5119 229/1563 [===>..........................] - ETA: 1s - loss: 1.3953 - accuracy: 0.5119 267/1563 [====>.........................] - ETA: 1s - loss: 1.3983 - accuracy: 0.5118 305/1563 [====>.........................] - ETA: 1s - loss: 1.3953 - accuracy: 0.5096 343/1563 [=====>........................] - ETA: 1s - loss: 1.3945 - accuracy: 0.5093 381/1563 [======>.......................] - ETA: 1s - loss: 1.3930 - accuracy: 0.5098 419/1563 [=======>......................] - ETA: 1s - loss: 1.3959 - accuracy: 0.5089 457/1563 [=======>......................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5096 495/1563 [========>.....................] - ETA: 1s - loss: 1.3923 - accuracy: 0.5106 533/1563 [=========>....................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5111 571/1563 [=========>....................] - ETA: 1s - loss: 1.3929 - accuracy: 0.5121 609/1563 [==========>...................] - ETA: 1s - loss: 1.3935 - accuracy: 0.5121 647/1563 [===========>..................] - ETA: 1s - loss: 1.3923 - accuracy: 0.5115 685/1563 [============>.................] - ETA: 1s - loss: 1.3915 - accuracy: 0.5119 723/1563 [============>.................] - ETA: 1s - loss: 1.3923 - accuracy: 0.5119 760/1563 [=============>................] - ETA: 1s - loss: 1.3916 - accuracy: 0.5122 798/1563 [==============>...............] - ETA: 1s - loss: 1.3907 - accuracy: 0.5125 836/1563 [===============>..............] - ETA: 0s - loss: 1.3894 - accuracy: 0.5127 874/1563 [===============>..............] - ETA: 0s - loss: 1.3878 - accuracy: 0.5137 912/1563 [================>.............] - ETA: 0s - loss: 1.3870 - accuracy: 0.5127 950/1563 [=================>............] - ETA: 0s - loss: 1.3873 - accuracy: 0.5119 988/1563 [=================>............] - ETA: 0s - loss: 1.3859 - accuracy: 0.5123 1026/1563 [==================>...........] - ETA: 0s - loss: 1.3880 - accuracy: 0.5113 1064/1563 [===================>..........] - ETA: 0s - loss: 1.3885 - accuracy: 0.5118 1102/1563 [====================>.........] - ETA: 0s - loss: 1.3889 - accuracy: 0.5121 1140/1563 [====================>.........] - ETA: 0s - loss: 1.3883 - accuracy: 0.5121 1178/1563 [=====================>........] - ETA: 0s - loss: 1.3887 - accuracy: 0.5122 1216/1563 [======================>.......] - ETA: 0s - loss: 1.3888 - accuracy: 0.5117 1254/1563 [=======================>......] - ETA: 0s - loss: 1.3880 - accuracy: 0.5121 1292/1563 [=======================>......] - ETA: 0s - loss: 1.3888 - accuracy: 0.5119 1330/1563 [========================>.....] - ETA: 0s - loss: 1.3889 - accuracy: 0.5123 1368/1563 [=========================>....] - ETA: 0s - loss: 1.3894 - accuracy: 0.5120 1406/1563 [=========================>....] - ETA: 0s - loss: 1.3895 - accuracy: 0.5124 1444/1563 [==========================>...] - ETA: 0s - loss: 1.3880 - accuracy: 0.5125 1481/1563 [===========================>..] - ETA: 0s - loss: 1.3885 - accuracy: 0.5118 1519/1563 [============================>.] - ETA: 0s - loss: 1.3874 - accuracy: 0.5120 1557/1563 [============================>.] - ETA: 0s - loss: 1.3871 - accuracy: 0.5121 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3874 - accuracy: 0.5119 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.5473 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.3901 - accuracy: 0.5232 77/1563 [>.............................] - ETA: 1s - loss: 1.3509 - accuracy: 0.5369 115/1563 [=>............................] - ETA: 1s - loss: 1.3575 - accuracy: 0.5274 153/1563 [=>............................] - ETA: 1s - loss: 1.3660 - accuracy: 0.5233 191/1563 [==>...........................] - ETA: 1s - loss: 1.3610 - accuracy: 0.5239 229/1563 [===>..........................] - ETA: 1s - loss: 1.3639 - accuracy: 0.5214 267/1563 [====>.........................] - ETA: 1s - loss: 1.3630 - accuracy: 0.5214 305/1563 [====>.........................] - ETA: 1s - loss: 1.3605 - accuracy: 0.5230 342/1563 [=====>........................] - ETA: 1s - loss: 1.3559 - accuracy: 0.5254 380/1563 [======>.......................] - ETA: 1s - loss: 1.3534 - accuracy: 0.5258 418/1563 [=======>......................] - ETA: 1s - loss: 1.3506 - accuracy: 0.5284 455/1563 [=======>......................] - ETA: 1s - loss: 1.3486 - accuracy: 0.5287 493/1563 [========>.....................] - ETA: 1s - loss: 1.3483 - accuracy: 0.5283 531/1563 [=========>....................] - ETA: 1s - loss: 1.3493 - accuracy: 0.5274 569/1563 [=========>....................] - ETA: 1s - loss: 1.3518 - accuracy: 0.5266 607/1563 [==========>...................] - ETA: 1s - loss: 1.3521 - accuracy: 0.5258 644/1563 [===========>..................] - ETA: 1s - loss: 1.3517 - accuracy: 0.5265 681/1563 [============>.................] - ETA: 1s - loss: 1.3507 - accuracy: 0.5273 719/1563 [============>.................] - ETA: 1s - loss: 1.3502 - accuracy: 0.5272 757/1563 [=============>................] - ETA: 1s - loss: 1.3511 - accuracy: 0.5261 795/1563 [==============>...............] - ETA: 1s - loss: 1.3506 - accuracy: 0.5265 833/1563 [==============>...............] - ETA: 0s - loss: 1.3519 - accuracy: 0.5255 871/1563 [===============>..............] - ETA: 0s - loss: 1.3542 - accuracy: 0.5250 909/1563 [================>.............] - ETA: 0s - loss: 1.3546 - accuracy: 0.5245 947/1563 [=================>............] - ETA: 0s - loss: 1.3539 - accuracy: 0.5245 984/1563 [=================>............] - ETA: 0s - loss: 1.3540 - accuracy: 0.5243 1022/1563 [==================>...........] - ETA: 0s - loss: 1.3551 - accuracy: 0.5234 1060/1563 [===================>..........] - ETA: 0s - loss: 1.3536 - accuracy: 0.5233 1098/1563 [====================>.........] - ETA: 0s - loss: 1.3533 - accuracy: 0.5229 1136/1563 [====================>.........] - ETA: 0s - loss: 1.3522 - accuracy: 0.5236 1174/1563 [=====================>........] - ETA: 0s - loss: 1.3512 - accuracy: 0.5235 1212/1563 [======================>.......] - ETA: 0s - loss: 1.3513 - accuracy: 0.5236 1250/1563 [======================>.......] - ETA: 0s - loss: 1.3513 - accuracy: 0.5239 1288/1563 [=======================>......] - ETA: 0s - loss: 1.3516 - accuracy: 0.5237 1326/1563 [========================>.....] - ETA: 0s - loss: 1.3507 - accuracy: 0.5246 1364/1563 [=========================>....] - ETA: 0s - loss: 1.3488 - accuracy: 0.5253 1402/1563 [=========================>....] - ETA: 0s - loss: 1.3491 - accuracy: 0.5251 1440/1563 [==========================>...] - ETA: 0s - loss: 1.3484 - accuracy: 0.5253 1478/1563 [===========================>..] - ETA: 0s - loss: 1.3481 - accuracy: 0.5257 1516/1563 [============================>.] - ETA: 0s - loss: 1.3487 - accuracy: 0.5257 1554/1563 [============================>.] - ETA: 0s - loss: 1.3486 - accuracy: 0.5258 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3490 - accuracy: 0.5255 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.1255 - accuracy: 0.6875 39/1563 [..............................] - ETA: 2s - loss: 1.2650 - accuracy: 0.5689 77/1563 [>.............................] - ETA: 1s - loss: 1.2755 - accuracy: 0.5536 115/1563 [=>............................] - ETA: 1s - loss: 1.2857 - accuracy: 0.5522 153/1563 [=>............................] - ETA: 1s - loss: 1.2877 - accuracy: 0.5502 191/1563 [==>...........................] - ETA: 1s - loss: 1.3020 - accuracy: 0.5429 228/1563 [===>..........................] - ETA: 1s - loss: 1.3025 - accuracy: 0.5433 266/1563 [====>.........................] - ETA: 1s - loss: 1.3033 - accuracy: 0.5422 304/1563 [====>.........................] - ETA: 1s - loss: 1.3035 - accuracy: 0.5395 342/1563 [=====>........................] - ETA: 1s - loss: 1.3050 - accuracy: 0.5380 380/1563 [======>.......................] - ETA: 1s - loss: 1.3048 - accuracy: 0.5367 418/1563 [=======>......................] - ETA: 1s - loss: 1.3051 - accuracy: 0.5386 456/1563 [=======>......................] - ETA: 1s - loss: 1.3123 - accuracy: 0.5368 493/1563 [========>.....................] - ETA: 1s - loss: 1.3120 - accuracy: 0.5392 531/1563 [=========>....................] - ETA: 1s - loss: 1.3116 - accuracy: 0.5396 568/1563 [=========>....................] - ETA: 1s - loss: 1.3137 - accuracy: 0.5384 606/1563 [==========>...................] - ETA: 1s - loss: 1.3145 - accuracy: 0.5375 643/1563 [===========>..................] - ETA: 1s - loss: 1.3154 - accuracy: 0.5366 681/1563 [============>.................] - ETA: 1s - loss: 1.3141 - accuracy: 0.5378 719/1563 [============>.................] - ETA: 1s - loss: 1.3160 - accuracy: 0.5379 757/1563 [=============>................] - ETA: 1s - loss: 1.3185 - accuracy: 0.5373 795/1563 [==============>...............] - ETA: 1s - loss: 1.3182 - accuracy: 0.5372 833/1563 [==============>...............] - ETA: 0s - loss: 1.3168 - accuracy: 0.5376 871/1563 [===============>..............] - ETA: 0s - loss: 1.3170 - accuracy: 0.5366 909/1563 [================>.............] - ETA: 0s - loss: 1.3165 - accuracy: 0.5369 947/1563 [=================>............] - ETA: 0s - loss: 1.3153 - accuracy: 0.5371 985/1563 [=================>............] - ETA: 0s - loss: 1.3151 - accuracy: 0.5372 1023/1563 [==================>...........] - ETA: 0s - loss: 1.3160 - accuracy: 0.5372 1061/1563 [===================>..........] - ETA: 0s - loss: 1.3167 - accuracy: 0.5367 1099/1563 [====================>.........] - ETA: 0s - loss: 1.3163 - accuracy: 0.5371 1137/1563 [====================>.........] - ETA: 0s - loss: 1.3138 - accuracy: 0.5377 1175/1563 [=====================>........] - ETA: 0s - loss: 1.3138 - accuracy: 0.5381 1213/1563 [======================>.......] - ETA: 0s - loss: 1.3156 - accuracy: 0.5375 1251/1563 [=======================>......] - ETA: 0s - loss: 1.3136 - accuracy: 0.5385 1289/1563 [=======================>......] - ETA: 0s - loss: 1.3117 - accuracy: 0.5390 1327/1563 [========================>.....] - ETA: 0s - loss: 1.3133 - accuracy: 0.5383 1365/1563 [=========================>....] - ETA: 0s - loss: 1.3120 - accuracy: 0.5393 1403/1563 [=========================>....] - ETA: 0s - loss: 1.3126 - accuracy: 0.5394 1441/1563 [==========================>...] - ETA: 0s - loss: 1.3138 - accuracy: 0.5384 1479/1563 [===========================>..] - ETA: 0s - loss: 1.3142 - accuracy: 0.5380 1516/1563 [============================>.] - ETA: 0s - loss: 1.3142 - accuracy: 0.5382 1554/1563 [============================>.] - ETA: 0s - loss: 1.3143 - accuracy: 0.5381 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3141 - accuracy: 0.5382 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.0196 - accuracy: 0.6250 39/1563 [..............................] - ETA: 2s - loss: 1.2485 - accuracy: 0.5617 77/1563 [>.............................] - ETA: 1s - loss: 1.2358 - accuracy: 0.5755 115/1563 [=>............................] - ETA: 1s - loss: 1.2665 - accuracy: 0.5584 152/1563 [=>............................] - ETA: 1s - loss: 1.2773 - accuracy: 0.5561 190/1563 [==>...........................] - ETA: 1s - loss: 1.2762 - accuracy: 0.5584 228/1563 [===>..........................] - ETA: 1s - loss: 1.2760 - accuracy: 0.5547 266/1563 [====>.........................] - ETA: 1s - loss: 1.2785 - accuracy: 0.5542 304/1563 [====>.........................] - ETA: 1s - loss: 1.2793 - accuracy: 0.5528 342/1563 [=====>........................] - ETA: 1s - loss: 1.2836 - accuracy: 0.5524 380/1563 [======>.......................] - ETA: 1s - loss: 1.2804 - accuracy: 0.5535 418/1563 [=======>......................] - ETA: 1s - loss: 1.2771 - accuracy: 0.5543 455/1563 [=======>......................] - ETA: 1s - loss: 1.2779 - accuracy: 0.5534 493/1563 [========>.....................] - ETA: 1s - loss: 1.2778 - accuracy: 0.5543 530/1563 [=========>....................] - ETA: 1s - loss: 1.2769 - accuracy: 0.5537 568/1563 [=========>....................] - ETA: 1s - loss: 1.2755 - accuracy: 0.5545 606/1563 [==========>...................] - ETA: 1s - loss: 1.2747 - accuracy: 0.5539 644/1563 [===========>..................] - ETA: 1s - loss: 1.2726 - accuracy: 0.5549 682/1563 [============>.................] - ETA: 1s - loss: 1.2738 - accuracy: 0.5546 720/1563 [============>.................] - ETA: 1s - loss: 1.2759 - accuracy: 0.5541 757/1563 [=============>................] - ETA: 1s - loss: 1.2770 - accuracy: 0.5531 794/1563 [==============>...............] - ETA: 1s - loss: 1.2755 - accuracy: 0.5530 832/1563 [==============>...............] - ETA: 0s - loss: 1.2748 - accuracy: 0.5529 870/1563 [===============>..............] - ETA: 0s - loss: 1.2746 - accuracy: 0.5532 908/1563 [================>.............] - ETA: 0s - loss: 1.2740 - accuracy: 0.5534 946/1563 [=================>............] - ETA: 0s - loss: 1.2738 - accuracy: 0.5536 984/1563 [=================>............] - ETA: 0s - loss: 1.2755 - accuracy: 0.5535 1022/1563 [==================>...........] - ETA: 0s - loss: 1.2757 - accuracy: 0.5536 1060/1563 [===================>..........] - ETA: 0s - loss: 1.2764 - accuracy: 0.5532 1098/1563 [====================>.........] - ETA: 0s - loss: 1.2760 - accuracy: 0.5534 1135/1563 [====================>.........] - ETA: 0s - loss: 1.2753 - accuracy: 0.5540 1173/1563 [=====================>........] - ETA: 0s - loss: 1.2788 - accuracy: 0.5530 1211/1563 [======================>.......] - ETA: 0s - loss: 1.2792 - accuracy: 0.5528 1249/1563 [======================>.......] - ETA: 0s - loss: 1.2817 - accuracy: 0.5519 1287/1563 [=======================>......] - ETA: 0s - loss: 1.2815 - accuracy: 0.5513 1325/1563 [========================>.....] - ETA: 0s - loss: 1.2813 - accuracy: 0.5513 1363/1563 [=========================>....] - ETA: 0s - loss: 1.2821 - accuracy: 0.5506 1400/1563 [=========================>....] - ETA: 0s - loss: 1.2830 - accuracy: 0.5506 1438/1563 [==========================>...] - ETA: 0s - loss: 1.2829 - accuracy: 0.5510 1475/1563 [===========================>..] - ETA: 0s - loss: 1.2811 - accuracy: 0.5517 1513/1563 [============================>.] - ETA: 0s - loss: 1.2808 - accuracy: 0.5517 1551/1563 [============================>.] - ETA: 0s - loss: 1.2810 - accuracy: 0.5515 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2813 - accuracy: 0.5512 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.2102 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.2898 - accuracy: 0.5369 76/1563 [>.............................] - ETA: 2s - loss: 1.2663 - accuracy: 0.5506 114/1563 [=>............................] - ETA: 1s - loss: 1.2675 - accuracy: 0.5515 152/1563 [=>............................] - ETA: 1s - loss: 1.2643 - accuracy: 0.5520 189/1563 [==>...........................] - ETA: 1s - loss: 1.2573 - accuracy: 0.5552 226/1563 [===>..........................] - ETA: 1s - loss: 1.2642 - accuracy: 0.5559 263/1563 [====>.........................] - ETA: 1s - loss: 1.2576 - accuracy: 0.5598 300/1563 [====>.........................] - ETA: 1s - loss: 1.2572 - accuracy: 0.5577 338/1563 [=====>........................] - ETA: 1s - loss: 1.2474 - accuracy: 0.5597 376/1563 [======>.......................] - ETA: 1s - loss: 1.2497 - accuracy: 0.5583 413/1563 [======>.......................] - ETA: 1s - loss: 1.2453 - accuracy: 0.5594 451/1563 [=======>......................] - ETA: 1s - loss: 1.2495 - accuracy: 0.5586 488/1563 [========>.....................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5572 526/1563 [=========>....................] - ETA: 1s - loss: 1.2532 - accuracy: 0.5568 563/1563 [=========>....................] - ETA: 1s - loss: 1.2477 - accuracy: 0.5587 601/1563 [==========>...................] - ETA: 1s - loss: 1.2474 - accuracy: 0.5577 638/1563 [===========>..................] - ETA: 1s - loss: 1.2458 - accuracy: 0.5586 675/1563 [===========>..................] - ETA: 1s - loss: 1.2442 - accuracy: 0.5595 713/1563 [============>.................] - ETA: 1s - loss: 1.2446 - accuracy: 0.5592 750/1563 [=============>................] - ETA: 1s - loss: 1.2453 - accuracy: 0.5596 788/1563 [==============>...............] - ETA: 1s - loss: 1.2451 - accuracy: 0.5606 826/1563 [==============>...............] - ETA: 0s - loss: 1.2436 - accuracy: 0.5609 864/1563 [===============>..............] - ETA: 0s - loss: 1.2425 - accuracy: 0.5608 902/1563 [================>.............] - ETA: 0s - loss: 1.2428 - accuracy: 0.5607 940/1563 [=================>............] - ETA: 0s - loss: 1.2433 - accuracy: 0.5597 977/1563 [=================>............] - ETA: 0s - loss: 1.2427 - accuracy: 0.5597 1014/1563 [==================>...........] - ETA: 0s - loss: 1.2454 - accuracy: 0.5591 1051/1563 [===================>..........] - ETA: 0s - loss: 1.2443 - accuracy: 0.5598 1088/1563 [===================>..........] - ETA: 0s - loss: 1.2441 - accuracy: 0.5597 1126/1563 [====================>.........] - ETA: 0s - loss: 1.2445 - accuracy: 0.5599 1164/1563 [=====================>........] - ETA: 0s - loss: 1.2442 - accuracy: 0.5608 1202/1563 [======================>.......] - ETA: 0s - loss: 1.2448 - accuracy: 0.5606 1240/1563 [======================>.......] - ETA: 0s - loss: 1.2438 - accuracy: 0.5610 1278/1563 [=======================>......] - ETA: 0s - loss: 1.2438 - accuracy: 0.5614 1316/1563 [========================>.....] - ETA: 0s - loss: 1.2432 - accuracy: 0.5617 1354/1563 [========================>.....] - ETA: 0s - loss: 1.2431 - accuracy: 0.5612 1391/1563 [=========================>....] - ETA: 0s - loss: 1.2448 - accuracy: 0.5610 1428/1563 [==========================>...] - ETA: 0s - loss: 1.2465 - accuracy: 0.5606 1466/1563 [===========================>..] - ETA: 0s - loss: 1.2468 - accuracy: 0.5605 1504/1563 [===========================>..] - ETA: 0s - loss: 1.2478 - accuracy: 0.5601 1542/1563 [============================>.] - ETA: 0s - loss: 1.2483 - accuracy: 0.5604 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2485 - accuracy: 0.5604 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 27.886109113693237 hydro03 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins4587736639406036172.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS