Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 97600 queued and waiting for resources srun: job 97600 has been allocated resources 2024-11-15 07:43:02.596504: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-15 07:43:03.859226: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-15 07:43:06.789458: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-15 07:43:06.790906: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-15 07:43:06.790913: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-15 07:43:14.014945: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-15 07:43:15.506781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-15 07:43:19.723791: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:18:55 - loss: 2.3972 - accuracy: 0.0625 37/1563 [..............................] - ETA: 2s - loss: 2.2543 - accuracy: 0.1630  74/1563 [>.............................] - ETA: 2s - loss: 2.1814 - accuracy: 0.1871 111/1563 [=>............................] - ETA: 2s - loss: 2.1344 - accuracy: 0.2120 148/1563 [=>............................] - ETA: 1s - loss: 2.0955 - accuracy: 0.2361 185/1563 [==>...........................] - ETA: 1s - loss: 2.0704 - accuracy: 0.2498 222/1563 [===>..........................] - ETA: 1s - loss: 2.0520 - accuracy: 0.2559 259/1563 [===>..........................] - ETA: 1s - loss: 2.0362 - accuracy: 0.2617 296/1563 [====>.........................] - ETA: 1s - loss: 2.0173 - accuracy: 0.2702 333/1563 [=====>........................] - ETA: 1s - loss: 2.0024 - accuracy: 0.2768 370/1563 [======>.......................] - ETA: 1s - loss: 1.9918 - accuracy: 0.2799 407/1563 [======>.......................] - ETA: 1s - loss: 1.9761 - accuracy: 0.2871 444/1563 [=======>......................] - ETA: 1s - loss: 1.9676 - accuracy: 0.2909 482/1563 [========>.....................] - ETA: 1s - loss: 1.9600 - accuracy: 0.2932 520/1563 [========>.....................] - ETA: 1s - loss: 1.9471 - accuracy: 0.2989 558/1563 [=========>....................] - ETA: 1s - loss: 1.9382 - accuracy: 0.3014 595/1563 [==========>...................] - ETA: 1s - loss: 1.9281 - accuracy: 0.3045 632/1563 [===========>..................] - ETA: 1s - loss: 1.9218 - accuracy: 0.3079 670/1563 [===========>..................] - ETA: 1s - loss: 1.9132 - accuracy: 0.3114 708/1563 [============>.................] - ETA: 1s - loss: 1.9088 - accuracy: 0.3137 746/1563 [=============>................] - ETA: 1s - loss: 1.9008 - accuracy: 0.3162 784/1563 [==============>...............] - ETA: 1s - loss: 1.8950 - accuracy: 0.3189 822/1563 [==============>...............] - ETA: 1s - loss: 1.8866 - accuracy: 0.3217 859/1563 [===============>..............] - ETA: 0s - loss: 1.8829 - accuracy: 0.3233 896/1563 [================>.............] - ETA: 0s - loss: 1.8779 - accuracy: 0.3254 934/1563 [================>.............] - ETA: 0s - loss: 1.8741 - accuracy: 0.3271 972/1563 [=================>............] - ETA: 0s - loss: 1.8688 - accuracy: 0.3298 1009/1563 [==================>...........] - ETA: 0s - loss: 1.8631 - accuracy: 0.3327 1047/1563 [===================>..........] - ETA: 0s - loss: 1.8590 - accuracy: 0.3339 1085/1563 [===================>..........] - ETA: 0s - loss: 1.8526 - accuracy: 0.3366 1123/1563 [====================>.........] - ETA: 0s - loss: 1.8482 - accuracy: 0.3388 1161/1563 [=====================>........] - ETA: 0s - loss: 1.8446 - accuracy: 0.3404 1199/1563 [======================>.......] - ETA: 0s - loss: 1.8412 - accuracy: 0.3420 1237/1563 [======================>.......] - ETA: 0s - loss: 1.8389 - accuracy: 0.3431 1275/1563 [=======================>......] - ETA: 0s - loss: 1.8353 - accuracy: 0.3443 1313/1563 [========================>.....] - ETA: 0s - loss: 1.8310 - accuracy: 0.3460 1350/1563 [========================>.....] - ETA: 0s - loss: 1.8279 - accuracy: 0.3471 1388/1563 [=========================>....] - ETA: 0s - loss: 1.8245 - accuracy: 0.3483 1426/1563 [==========================>...] - ETA: 0s - loss: 1.8214 - accuracy: 0.3500 1464/1563 [===========================>..] - ETA: 0s - loss: 1.8189 - accuracy: 0.3508 1502/1563 [===========================>..] - ETA: 0s - loss: 1.8168 - accuracy: 0.3519 1540/1563 [============================>.] - ETA: 0s - loss: 1.8135 - accuracy: 0.3532 1563/1563 [==============================] - 5s 1ms/step - loss: 1.8119 - accuracy: 0.3536 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.8632 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.6877 - accuracy: 0.4111 77/1563 [>.............................] - ETA: 2s - loss: 1.6872 - accuracy: 0.4095 115/1563 [=>............................] - ETA: 1s - loss: 1.6735 - accuracy: 0.4117 153/1563 [=>............................] - ETA: 1s - loss: 1.6661 - accuracy: 0.4126 191/1563 [==>...........................] - ETA: 1s - loss: 1.6680 - accuracy: 0.4102 229/1563 [===>..........................] - ETA: 1s - loss: 1.6626 - accuracy: 0.4106 267/1563 [====>.........................] - ETA: 1s - loss: 1.6660 - accuracy: 0.4105 305/1563 [====>.........................] - ETA: 1s - loss: 1.6658 - accuracy: 0.4107 343/1563 [=====>........................] - ETA: 1s - loss: 1.6656 - accuracy: 0.4155 381/1563 [======>.......................] - ETA: 1s - loss: 1.6642 - accuracy: 0.4147 419/1563 [=======>......................] - ETA: 1s - loss: 1.6593 - accuracy: 0.4164 457/1563 [=======>......................] - ETA: 1s - loss: 1.6556 - accuracy: 0.4165 495/1563 [========>.....................] - ETA: 1s - loss: 1.6536 - accuracy: 0.4162 533/1563 [=========>....................] - ETA: 1s - loss: 1.6532 - accuracy: 0.4153 571/1563 [=========>....................] - ETA: 1s - loss: 1.6530 - accuracy: 0.4150 609/1563 [==========>...................] - ETA: 1s - loss: 1.6511 - accuracy: 0.4150 647/1563 [===========>..................] - ETA: 1s - loss: 1.6507 - accuracy: 0.4144 685/1563 [============>.................] - ETA: 1s - loss: 1.6488 - accuracy: 0.4149 723/1563 [============>.................] - ETA: 1s - loss: 1.6468 - accuracy: 0.4166 761/1563 [=============>................] - ETA: 1s - loss: 1.6449 - accuracy: 0.4187 799/1563 [==============>...............] - ETA: 1s - loss: 1.6432 - accuracy: 0.4192 837/1563 [===============>..............] - ETA: 0s - loss: 1.6412 - accuracy: 0.4198 875/1563 [===============>..............] - ETA: 0s - loss: 1.6396 - accuracy: 0.4203 913/1563 [================>.............] - ETA: 0s - loss: 1.6375 - accuracy: 0.4206 950/1563 [=================>............] - ETA: 0s - loss: 1.6366 - accuracy: 0.4204 988/1563 [=================>............] - ETA: 0s - loss: 1.6355 - accuracy: 0.4208 1026/1563 [==================>...........] - ETA: 0s - loss: 1.6328 - accuracy: 0.4223 1064/1563 [===================>..........] - ETA: 0s - loss: 1.6314 - accuracy: 0.4233 1102/1563 [====================>.........] - ETA: 0s - loss: 1.6311 - accuracy: 0.4231 1139/1563 [====================>.........] - ETA: 0s - loss: 1.6302 - accuracy: 0.4236 1177/1563 [=====================>........] - ETA: 0s - loss: 1.6303 - accuracy: 0.4240 1215/1563 [======================>.......] - ETA: 0s - loss: 1.6302 - accuracy: 0.4246 1253/1563 [=======================>......] - ETA: 0s - loss: 1.6287 - accuracy: 0.4256 1290/1563 [=======================>......] - ETA: 0s - loss: 1.6278 - accuracy: 0.4256 1328/1563 [========================>.....] - ETA: 0s - loss: 1.6275 - accuracy: 0.4259 1365/1563 [=========================>....] - ETA: 0s - loss: 1.6267 - accuracy: 0.4263 1403/1563 [=========================>....] - ETA: 0s - loss: 1.6255 - accuracy: 0.4268 1440/1563 [==========================>...] - ETA: 0s - loss: 1.6251 - accuracy: 0.4269 1477/1563 [===========================>..] - ETA: 0s - loss: 1.6241 - accuracy: 0.4274 1515/1563 [============================>.] - ETA: 0s - loss: 1.6229 - accuracy: 0.4279 1552/1563 [============================>.] - ETA: 0s - loss: 1.6215 - accuracy: 0.4282 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6208 - accuracy: 0.4284 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.7690 - accuracy: 0.3750 38/1563 [..............................] - ETA: 2s - loss: 1.5623 - accuracy: 0.4474 75/1563 [>.............................] - ETA: 2s - loss: 1.5473 - accuracy: 0.4538 112/1563 [=>............................] - ETA: 1s - loss: 1.5540 - accuracy: 0.4489 149/1563 [=>............................] - ETA: 1s - loss: 1.5508 - accuracy: 0.4503 187/1563 [==>...........................] - ETA: 1s - loss: 1.5553 - accuracy: 0.4480 225/1563 [===>..........................] - ETA: 1s - loss: 1.5483 - accuracy: 0.4490 263/1563 [====>.........................] - ETA: 1s - loss: 1.5522 - accuracy: 0.4494 301/1563 [====>.........................] - ETA: 1s - loss: 1.5469 - accuracy: 0.4516 338/1563 [=====>........................] - ETA: 1s - loss: 1.5501 - accuracy: 0.4503 376/1563 [======>.......................] - ETA: 1s - loss: 1.5554 - accuracy: 0.4495 414/1563 [======>.......................] - ETA: 1s - loss: 1.5554 - accuracy: 0.4506 452/1563 [=======>......................] - ETA: 1s - loss: 1.5528 - accuracy: 0.4520 489/1563 [========>.....................] - ETA: 1s - loss: 1.5523 - accuracy: 0.4523 527/1563 [=========>....................] - ETA: 1s - loss: 1.5555 - accuracy: 0.4509 565/1563 [=========>....................] - ETA: 1s - loss: 1.5576 - accuracy: 0.4499 603/1563 [==========>...................] - ETA: 1s - loss: 1.5583 - accuracy: 0.4503 640/1563 [===========>..................] - ETA: 1s - loss: 1.5568 - accuracy: 0.4515 677/1563 [===========>..................] - ETA: 1s - loss: 1.5560 - accuracy: 0.4511 714/1563 [============>.................] - ETA: 1s - loss: 1.5552 - accuracy: 0.4516 751/1563 [=============>................] - ETA: 1s - loss: 1.5527 - accuracy: 0.4521 788/1563 [==============>...............] - ETA: 1s - loss: 1.5545 - accuracy: 0.4505 826/1563 [==============>...............] - ETA: 0s - loss: 1.5536 - accuracy: 0.4504 864/1563 [===============>..............] - ETA: 0s - loss: 1.5532 - accuracy: 0.4510 901/1563 [================>.............] - ETA: 0s - loss: 1.5528 - accuracy: 0.4514 938/1563 [=================>............] - ETA: 0s - loss: 1.5526 - accuracy: 0.4513 975/1563 [=================>............] - ETA: 0s - loss: 1.5518 - accuracy: 0.4519 1013/1563 [==================>...........] - ETA: 0s - loss: 1.5532 - accuracy: 0.4511 1051/1563 [===================>..........] - ETA: 0s - loss: 1.5510 - accuracy: 0.4520 1089/1563 [===================>..........] - ETA: 0s - loss: 1.5504 - accuracy: 0.4527 1127/1563 [====================>.........] - ETA: 0s - loss: 1.5496 - accuracy: 0.4526 1165/1563 [=====================>........] - ETA: 0s - loss: 1.5495 - accuracy: 0.4532 1203/1563 [======================>.......] - ETA: 0s - loss: 1.5499 - accuracy: 0.4528 1241/1563 [======================>.......] - ETA: 0s - loss: 1.5489 - accuracy: 0.4534 1278/1563 [=======================>......] - ETA: 0s - loss: 1.5482 - accuracy: 0.4534 1315/1563 [========================>.....] - ETA: 0s - loss: 1.5471 - accuracy: 0.4538 1353/1563 [========================>.....] - ETA: 0s - loss: 1.5457 - accuracy: 0.4545 1391/1563 [=========================>....] - ETA: 0s - loss: 1.5443 - accuracy: 0.4554 1429/1563 [==========================>...] - ETA: 0s - loss: 1.5440 - accuracy: 0.4555 1467/1563 [===========================>..] - ETA: 0s - loss: 1.5433 - accuracy: 0.4555 1504/1563 [===========================>..] - ETA: 0s - loss: 1.5424 - accuracy: 0.4557 1541/1563 [============================>.] - ETA: 0s - loss: 1.5420 - accuracy: 0.4552 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5416 - accuracy: 0.4556 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.2655 - accuracy: 0.5000 36/1563 [..............................] - ETA: 2s - loss: 1.4988 - accuracy: 0.4800 73/1563 [>.............................] - ETA: 2s - loss: 1.4953 - accuracy: 0.4717 110/1563 [=>............................] - ETA: 2s - loss: 1.5104 - accuracy: 0.4622 148/1563 [=>............................] - ETA: 1s - loss: 1.5110 - accuracy: 0.4641 186/1563 [==>...........................] - ETA: 1s - loss: 1.5104 - accuracy: 0.4619 224/1563 [===>..........................] - ETA: 1s - loss: 1.5097 - accuracy: 0.4644 261/1563 [====>.........................] - ETA: 1s - loss: 1.5068 - accuracy: 0.4671 298/1563 [====>.........................] - ETA: 1s - loss: 1.5077 - accuracy: 0.4671 335/1563 [=====>........................] - ETA: 1s - loss: 1.5038 - accuracy: 0.4695 372/1563 [======>.......................] - ETA: 1s - loss: 1.5005 - accuracy: 0.4707 409/1563 [======>.......................] - ETA: 1s - loss: 1.4995 - accuracy: 0.4700 447/1563 [=======>......................] - ETA: 1s - loss: 1.5031 - accuracy: 0.4700 485/1563 [========>.....................] - ETA: 1s - loss: 1.4988 - accuracy: 0.4704 523/1563 [=========>....................] - ETA: 1s - loss: 1.5011 - accuracy: 0.4699 560/1563 [=========>....................] - ETA: 1s - loss: 1.4992 - accuracy: 0.4706 598/1563 [==========>...................] - ETA: 1s - loss: 1.4993 - accuracy: 0.4705 635/1563 [===========>..................] - ETA: 1s - loss: 1.4953 - accuracy: 0.4721 672/1563 [===========>..................] - ETA: 1s - loss: 1.4943 - accuracy: 0.4719 710/1563 [============>.................] - ETA: 1s - loss: 1.4909 - accuracy: 0.4735 748/1563 [=============>................] - ETA: 1s - loss: 1.4904 - accuracy: 0.4743 786/1563 [==============>...............] - ETA: 1s - loss: 1.4911 - accuracy: 0.4744 824/1563 [==============>...............] - ETA: 1s - loss: 1.4927 - accuracy: 0.4743 861/1563 [===============>..............] - ETA: 0s - loss: 1.4939 - accuracy: 0.4743 898/1563 [================>.............] - ETA: 0s - loss: 1.4933 - accuracy: 0.4749 934/1563 [================>.............] - ETA: 0s - loss: 1.4925 - accuracy: 0.4750 971/1563 [=================>............] - ETA: 0s - loss: 1.4894 - accuracy: 0.4758 1008/1563 [==================>...........] - ETA: 0s - loss: 1.4897 - accuracy: 0.4750 1046/1563 [===================>..........] - ETA: 0s - loss: 1.4896 - accuracy: 0.4746 1084/1563 [===================>..........] - ETA: 0s - loss: 1.4898 - accuracy: 0.4745 1122/1563 [====================>.........] - ETA: 0s - loss: 1.4903 - accuracy: 0.4745 1159/1563 [=====================>........] - ETA: 0s - loss: 1.4888 - accuracy: 0.4752 1197/1563 [=====================>........] - ETA: 0s - loss: 1.4887 - accuracy: 0.4751 1234/1563 [======================>.......] - ETA: 0s - loss: 1.4877 - accuracy: 0.4752 1271/1563 [=======================>......] - ETA: 0s - loss: 1.4866 - accuracy: 0.4760 1308/1563 [========================>.....] - ETA: 0s - loss: 1.4856 - accuracy: 0.4765 1346/1563 [========================>.....] - ETA: 0s - loss: 1.4855 - accuracy: 0.4769 1384/1563 [=========================>....] - ETA: 0s - loss: 1.4851 - accuracy: 0.4769 1421/1563 [==========================>...] - ETA: 0s - loss: 1.4843 - accuracy: 0.4773 1458/1563 [==========================>...] - ETA: 0s - loss: 1.4841 - accuracy: 0.4775 1495/1563 [===========================>..] - ETA: 0s - loss: 1.4840 - accuracy: 0.4774 1532/1563 [============================>.] - ETA: 0s - loss: 1.4838 - accuracy: 0.4776 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4835 - accuracy: 0.4779 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.2558 - accuracy: 0.5938 38/1563 [..............................] - ETA: 2s - loss: 1.4767 - accuracy: 0.4844 76/1563 [>.............................] - ETA: 2s - loss: 1.4678 - accuracy: 0.4831 113/1563 [=>............................] - ETA: 1s - loss: 1.4511 - accuracy: 0.4959 151/1563 [=>............................] - ETA: 1s - loss: 1.4471 - accuracy: 0.4934 189/1563 [==>...........................] - ETA: 1s - loss: 1.4404 - accuracy: 0.4998 226/1563 [===>..........................] - ETA: 1s - loss: 1.4432 - accuracy: 0.4964 263/1563 [====>.........................] - ETA: 1s - loss: 1.4417 - accuracy: 0.4952 300/1563 [====>.........................] - ETA: 1s - loss: 1.4366 - accuracy: 0.4981 337/1563 [=====>........................] - ETA: 1s - loss: 1.4330 - accuracy: 0.5010 375/1563 [======>.......................] - ETA: 1s - loss: 1.4381 - accuracy: 0.4972 413/1563 [======>.......................] - ETA: 1s - loss: 1.4346 - accuracy: 0.4989 451/1563 [=======>......................] - ETA: 1s - loss: 1.4333 - accuracy: 0.4990 489/1563 [========>.....................] - ETA: 1s - loss: 1.4355 - accuracy: 0.4980 527/1563 [=========>....................] - ETA: 1s - loss: 1.4309 - accuracy: 0.4993 565/1563 [=========>....................] - ETA: 1s - loss: 1.4311 - accuracy: 0.4997 602/1563 [==========>...................] - ETA: 1s - loss: 1.4311 - accuracy: 0.5002 639/1563 [===========>..................] - ETA: 1s - loss: 1.4299 - accuracy: 0.5005 677/1563 [===========>..................] - ETA: 1s - loss: 1.4312 - accuracy: 0.4991 715/1563 [============>.................] - ETA: 1s - loss: 1.4303 - accuracy: 0.4989 753/1563 [=============>................] - ETA: 1s - loss: 1.4318 - accuracy: 0.4986 791/1563 [==============>...............] - ETA: 1s - loss: 1.4317 - accuracy: 0.4986 828/1563 [==============>...............] - ETA: 0s - loss: 1.4312 - accuracy: 0.4985 865/1563 [===============>..............] - ETA: 0s - loss: 1.4330 - accuracy: 0.4979 903/1563 [================>.............] - ETA: 0s - loss: 1.4332 - accuracy: 0.4981 940/1563 [=================>............] - ETA: 0s - loss: 1.4326 - accuracy: 0.4982 977/1563 [=================>............] - ETA: 0s - loss: 1.4335 - accuracy: 0.4972 1015/1563 [==================>...........] - ETA: 0s - loss: 1.4337 - accuracy: 0.4968 1052/1563 [===================>..........] - ETA: 0s - loss: 1.4333 - accuracy: 0.4967 1090/1563 [===================>..........] - ETA: 0s - loss: 1.4344 - accuracy: 0.4962 1127/1563 [====================>.........] - ETA: 0s - loss: 1.4342 - accuracy: 0.4961 1165/1563 [=====================>........] - ETA: 0s - loss: 1.4346 - accuracy: 0.4950 1203/1563 [======================>.......] - ETA: 0s - loss: 1.4334 - accuracy: 0.4956 1240/1563 [======================>.......] - ETA: 0s - loss: 1.4341 - accuracy: 0.4958 1277/1563 [=======================>......] - ETA: 0s - loss: 1.4346 - accuracy: 0.4951 1314/1563 [========================>.....] - ETA: 0s - loss: 1.4343 - accuracy: 0.4949 1351/1563 [========================>.....] - ETA: 0s - loss: 1.4334 - accuracy: 0.4957 1388/1563 [=========================>....] - ETA: 0s - loss: 1.4341 - accuracy: 0.4953 1425/1563 [==========================>...] - ETA: 0s - loss: 1.4332 - accuracy: 0.4959 1462/1563 [===========================>..] - ETA: 0s - loss: 1.4322 - accuracy: 0.4965 1499/1563 [===========================>..] - ETA: 0s - loss: 1.4322 - accuracy: 0.4965 1536/1563 [============================>.] - ETA: 0s - loss: 1.4319 - accuracy: 0.4963 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4325 - accuracy: 0.4961 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.4330 - accuracy: 0.5312 38/1563 [..............................] - ETA: 2s - loss: 1.3813 - accuracy: 0.5255 75/1563 [>.............................] - ETA: 2s - loss: 1.3706 - accuracy: 0.5296 112/1563 [=>............................] - ETA: 1s - loss: 1.3790 - accuracy: 0.5179 149/1563 [=>............................] - ETA: 1s - loss: 1.3944 - accuracy: 0.5117 186/1563 [==>...........................] - ETA: 1s - loss: 1.4082 - accuracy: 0.5076 223/1563 [===>..........................] - ETA: 1s - loss: 1.4004 - accuracy: 0.5083 260/1563 [===>..........................] - ETA: 1s - loss: 1.3991 - accuracy: 0.5119 297/1563 [====>.........................] - ETA: 1s - loss: 1.3989 - accuracy: 0.5123 334/1563 [=====>........................] - ETA: 1s - loss: 1.3945 - accuracy: 0.5116 371/1563 [======>.......................] - ETA: 1s - loss: 1.3974 - accuracy: 0.5104 408/1563 [======>.......................] - ETA: 1s - loss: 1.3940 - accuracy: 0.5120 445/1563 [=======>......................] - ETA: 1s - loss: 1.3944 - accuracy: 0.5115 482/1563 [========>.....................] - ETA: 1s - loss: 1.3930 - accuracy: 0.5117 519/1563 [========>.....................] - ETA: 1s - loss: 1.3899 - accuracy: 0.5117 556/1563 [=========>....................] - ETA: 1s - loss: 1.3923 - accuracy: 0.5108 593/1563 [==========>...................] - ETA: 1s - loss: 1.3917 - accuracy: 0.5119 630/1563 [===========>..................] - ETA: 1s - loss: 1.3940 - accuracy: 0.5099 667/1563 [===========>..................] - ETA: 1s - loss: 1.3941 - accuracy: 0.5097 704/1563 [============>.................] - ETA: 1s - loss: 1.3925 - accuracy: 0.5097 741/1563 [=============>................] - ETA: 1s - loss: 1.3933 - accuracy: 0.5090 778/1563 [=============>................] - ETA: 1s - loss: 1.3927 - accuracy: 0.5092 815/1563 [==============>...............] - ETA: 1s - loss: 1.3936 - accuracy: 0.5100 852/1563 [===============>..............] - ETA: 0s - loss: 1.3943 - accuracy: 0.5096 889/1563 [================>.............] - ETA: 0s - loss: 1.3924 - accuracy: 0.5102 926/1563 [================>.............] - ETA: 0s - loss: 1.3920 - accuracy: 0.5101 963/1563 [=================>............] - ETA: 0s - loss: 1.3914 - accuracy: 0.5108 1000/1563 [==================>...........] - ETA: 0s - loss: 1.3899 - accuracy: 0.5119 1037/1563 [==================>...........] - ETA: 0s - loss: 1.3922 - accuracy: 0.5107 1074/1563 [===================>..........] - ETA: 0s - loss: 1.3925 - accuracy: 0.5106 1111/1563 [====================>.........] - ETA: 0s - loss: 1.3927 - accuracy: 0.5103 1148/1563 [=====================>........] - ETA: 0s - loss: 1.3933 - accuracy: 0.5098 1185/1563 [=====================>........] - ETA: 0s - loss: 1.3930 - accuracy: 0.5095 1222/1563 [======================>.......] - ETA: 0s - loss: 1.3927 - accuracy: 0.5099 1259/1563 [=======================>......] - ETA: 0s - loss: 1.3919 - accuracy: 0.5104 1296/1563 [=======================>......] - ETA: 0s - loss: 1.3921 - accuracy: 0.5098 1333/1563 [========================>.....] - ETA: 0s - loss: 1.3914 - accuracy: 0.5100 1370/1563 [=========================>....] - ETA: 0s - loss: 1.3912 - accuracy: 0.5100 1407/1563 [==========================>...] - ETA: 0s - loss: 1.3916 - accuracy: 0.5098 1444/1563 [==========================>...] - ETA: 0s - loss: 1.3917 - accuracy: 0.5096 1481/1563 [===========================>..] - ETA: 0s - loss: 1.3905 - accuracy: 0.5099 1518/1563 [============================>.] - ETA: 0s - loss: 1.3907 - accuracy: 0.5093 1555/1563 [============================>.] - ETA: 0s - loss: 1.3885 - accuracy: 0.5102 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3880 - accuracy: 0.5104 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.6578 - accuracy: 0.4375 38/1563 [..............................] - ETA: 2s - loss: 1.3997 - accuracy: 0.5016 75/1563 [>.............................] - ETA: 2s - loss: 1.3911 - accuracy: 0.5196 112/1563 [=>............................] - ETA: 1s - loss: 1.3688 - accuracy: 0.5262 149/1563 [=>............................] - ETA: 1s - loss: 1.3620 - accuracy: 0.5277 186/1563 [==>...........................] - ETA: 1s - loss: 1.3532 - accuracy: 0.5302 223/1563 [===>..........................] - ETA: 1s - loss: 1.3581 - accuracy: 0.5241 260/1563 [===>..........................] - ETA: 1s - loss: 1.3512 - accuracy: 0.5262 297/1563 [====>.........................] - ETA: 1s - loss: 1.3529 - accuracy: 0.5263 334/1563 [=====>........................] - ETA: 1s - loss: 1.3476 - accuracy: 0.5275 371/1563 [======>.......................] - ETA: 1s - loss: 1.3493 - accuracy: 0.5277 408/1563 [======>.......................] - ETA: 1s - loss: 1.3529 - accuracy: 0.5272 445/1563 [=======>......................] - ETA: 1s - loss: 1.3525 - accuracy: 0.5276 482/1563 [========>.....................] - ETA: 1s - loss: 1.3495 - accuracy: 0.5277 519/1563 [========>.....................] - ETA: 1s - loss: 1.3497 - accuracy: 0.5273 556/1563 [=========>....................] - ETA: 1s - loss: 1.3465 - accuracy: 0.5280 593/1563 [==========>...................] - ETA: 1s - loss: 1.3479 - accuracy: 0.5267 630/1563 [===========>..................] - ETA: 1s - loss: 1.3508 - accuracy: 0.5265 667/1563 [===========>..................] - ETA: 1s - loss: 1.3491 - accuracy: 0.5259 704/1563 [============>.................] - ETA: 1s - loss: 1.3511 - accuracy: 0.5255 741/1563 [=============>................] - ETA: 1s - loss: 1.3523 - accuracy: 0.5252 778/1563 [=============>................] - ETA: 1s - loss: 1.3526 - accuracy: 0.5245 815/1563 [==============>...............] - ETA: 1s - loss: 1.3539 - accuracy: 0.5236 852/1563 [===============>..............] - ETA: 0s - loss: 1.3532 - accuracy: 0.5235 889/1563 [================>.............] - ETA: 0s - loss: 1.3529 - accuracy: 0.5235 926/1563 [================>.............] - ETA: 0s - loss: 1.3529 - accuracy: 0.5240 963/1563 [=================>............] - ETA: 0s - loss: 1.3526 - accuracy: 0.5245 1000/1563 [==================>...........] - ETA: 0s - loss: 1.3522 - accuracy: 0.5243 1037/1563 [==================>...........] - ETA: 0s - loss: 1.3532 - accuracy: 0.5243 1074/1563 [===================>..........] - ETA: 0s - loss: 1.3529 - accuracy: 0.5242 1111/1563 [====================>.........] - ETA: 0s - loss: 1.3508 - accuracy: 0.5252 1148/1563 [=====================>........] - ETA: 0s - loss: 1.3518 - accuracy: 0.5252 1185/1563 [=====================>........] - ETA: 0s - loss: 1.3525 - accuracy: 0.5246 1222/1563 [======================>.......] - ETA: 0s - loss: 1.3536 - accuracy: 0.5242 1259/1563 [=======================>......] - ETA: 0s - loss: 1.3534 - accuracy: 0.5239 1296/1563 [=======================>......] - ETA: 0s - loss: 1.3543 - accuracy: 0.5239 1333/1563 [========================>.....] - ETA: 0s - loss: 1.3539 - accuracy: 0.5240 1370/1563 [=========================>....] - ETA: 0s - loss: 1.3521 - accuracy: 0.5247 1407/1563 [==========================>...] - ETA: 0s - loss: 1.3521 - accuracy: 0.5246 1444/1563 [==========================>...] - ETA: 0s - loss: 1.3512 - accuracy: 0.5248 1481/1563 [===========================>..] - ETA: 0s - loss: 1.3513 - accuracy: 0.5248 1518/1563 [============================>.] - ETA: 0s - loss: 1.3518 - accuracy: 0.5245 1555/1563 [============================>.] - ETA: 0s - loss: 1.3513 - accuracy: 0.5245 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3509 - accuracy: 0.5246 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.5302 - accuracy: 0.4062 38/1563 [..............................] - ETA: 2s - loss: 1.3312 - accuracy: 0.5403 75/1563 [>.............................] - ETA: 2s - loss: 1.3183 - accuracy: 0.5383 112/1563 [=>............................] - ETA: 1s - loss: 1.3239 - accuracy: 0.5346 149/1563 [=>............................] - ETA: 1s - loss: 1.3146 - accuracy: 0.5380 186/1563 [==>...........................] - ETA: 1s - loss: 1.3192 - accuracy: 0.5361 223/1563 [===>..........................] - ETA: 1s - loss: 1.3192 - accuracy: 0.5348 260/1563 [===>..........................] - ETA: 1s - loss: 1.3214 - accuracy: 0.5339 297/1563 [====>.........................] - ETA: 1s - loss: 1.3159 - accuracy: 0.5356 334/1563 [=====>........................] - ETA: 1s - loss: 1.3171 - accuracy: 0.5354 371/1563 [======>.......................] - ETA: 1s - loss: 1.3154 - accuracy: 0.5356 408/1563 [======>.......................] - ETA: 1s - loss: 1.3152 - accuracy: 0.5366 445/1563 [=======>......................] - ETA: 1s - loss: 1.3156 - accuracy: 0.5368 482/1563 [========>.....................] - ETA: 1s - loss: 1.3139 - accuracy: 0.5378 519/1563 [========>.....................] - ETA: 1s - loss: 1.3145 - accuracy: 0.5387 556/1563 [=========>....................] - ETA: 1s - loss: 1.3167 - accuracy: 0.5379 593/1563 [==========>...................] - ETA: 1s - loss: 1.3169 - accuracy: 0.5385 630/1563 [===========>..................] - ETA: 1s - loss: 1.3159 - accuracy: 0.5394 667/1563 [===========>..................] - ETA: 1s - loss: 1.3190 - accuracy: 0.5385 704/1563 [============>.................] - ETA: 1s - loss: 1.3171 - accuracy: 0.5387 741/1563 [=============>................] - ETA: 1s - loss: 1.3175 - accuracy: 0.5382 778/1563 [=============>................] - ETA: 1s - loss: 1.3161 - accuracy: 0.5381 815/1563 [==============>...............] - ETA: 1s - loss: 1.3141 - accuracy: 0.5382 852/1563 [===============>..............] - ETA: 0s - loss: 1.3145 - accuracy: 0.5381 889/1563 [================>.............] - ETA: 0s - loss: 1.3133 - accuracy: 0.5384 926/1563 [================>.............] - ETA: 0s - loss: 1.3149 - accuracy: 0.5381 963/1563 [=================>............] - ETA: 0s - loss: 1.3123 - accuracy: 0.5395 1000/1563 [==================>...........] - ETA: 0s - loss: 1.3123 - accuracy: 0.5391 1037/1563 [==================>...........] - ETA: 0s - loss: 1.3132 - accuracy: 0.5392 1074/1563 [===================>..........] - ETA: 0s - loss: 1.3153 - accuracy: 0.5388 1111/1563 [====================>.........] - ETA: 0s - loss: 1.3178 - accuracy: 0.5380 1148/1563 [=====================>........] - ETA: 0s - loss: 1.3176 - accuracy: 0.5376 1185/1563 [=====================>........] - ETA: 0s - loss: 1.3186 - accuracy: 0.5368 1222/1563 [======================>.......] - ETA: 0s - loss: 1.3172 - accuracy: 0.5371 1259/1563 [=======================>......] - ETA: 0s - loss: 1.3180 - accuracy: 0.5359 1296/1563 [=======================>......] - ETA: 0s - loss: 1.3176 - accuracy: 0.5363 1333/1563 [========================>.....] - ETA: 0s - loss: 1.3178 - accuracy: 0.5359 1370/1563 [=========================>....] - ETA: 0s - loss: 1.3158 - accuracy: 0.5363 1407/1563 [==========================>...] - ETA: 0s - loss: 1.3151 - accuracy: 0.5365 1444/1563 [==========================>...] - ETA: 0s - loss: 1.3157 - accuracy: 0.5363 1481/1563 [===========================>..] - ETA: 0s - loss: 1.3148 - accuracy: 0.5368 1518/1563 [============================>.] - ETA: 0s - loss: 1.3129 - accuracy: 0.5374 1555/1563 [============================>.] - ETA: 0s - loss: 1.3139 - accuracy: 0.5370 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3142 - accuracy: 0.5369 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.1082 - accuracy: 0.5938 38/1563 [..............................] - ETA: 2s - loss: 1.2596 - accuracy: 0.5633 75/1563 [>.............................] - ETA: 2s - loss: 1.2809 - accuracy: 0.5537 112/1563 [=>............................] - ETA: 1s - loss: 1.2814 - accuracy: 0.5511 149/1563 [=>............................] - ETA: 1s - loss: 1.2837 - accuracy: 0.5539 186/1563 [==>...........................] - ETA: 1s - loss: 1.2848 - accuracy: 0.5521 223/1563 [===>..........................] - ETA: 1s - loss: 1.2735 - accuracy: 0.5563 260/1563 [===>..........................] - ETA: 1s - loss: 1.2738 - accuracy: 0.5561 297/1563 [====>.........................] - ETA: 1s - loss: 1.2842 - accuracy: 0.5502 334/1563 [=====>........................] - ETA: 1s - loss: 1.2810 - accuracy: 0.5523 371/1563 [======>.......................] - ETA: 1s - loss: 1.2882 - accuracy: 0.5484 408/1563 [======>.......................] - ETA: 1s - loss: 1.2899 - accuracy: 0.5489 445/1563 [=======>......................] - ETA: 1s - loss: 1.2898 - accuracy: 0.5490 481/1563 [========>.....................] - ETA: 1s - loss: 1.2862 - accuracy: 0.5512 519/1563 [========>.....................] - ETA: 1s - loss: 1.2872 - accuracy: 0.5499 556/1563 [=========>....................] - ETA: 1s - loss: 1.2876 - accuracy: 0.5496 593/1563 [==========>...................] - ETA: 1s - loss: 1.2890 - accuracy: 0.5505 630/1563 [===========>..................] - ETA: 1s - loss: 1.2906 - accuracy: 0.5499 668/1563 [===========>..................] - ETA: 1s - loss: 1.2918 - accuracy: 0.5491 706/1563 [============>.................] - ETA: 1s - loss: 1.2917 - accuracy: 0.5498 744/1563 [=============>................] - ETA: 1s - loss: 1.2911 - accuracy: 0.5499 782/1563 [==============>...............] - ETA: 1s - loss: 1.2886 - accuracy: 0.5508 820/1563 [==============>...............] - ETA: 1s - loss: 1.2884 - accuracy: 0.5508 858/1563 [===============>..............] - ETA: 0s - loss: 1.2893 - accuracy: 0.5498 896/1563 [================>.............] - ETA: 0s - loss: 1.2900 - accuracy: 0.5495 934/1563 [================>.............] - ETA: 0s - loss: 1.2903 - accuracy: 0.5492 972/1563 [=================>............] - ETA: 0s - loss: 1.2895 - accuracy: 0.5495 1010/1563 [==================>...........] - ETA: 0s - loss: 1.2901 - accuracy: 0.5492 1048/1563 [===================>..........] - ETA: 0s - loss: 1.2893 - accuracy: 0.5491 1086/1563 [===================>..........] - ETA: 0s - loss: 1.2894 - accuracy: 0.5487 1124/1563 [====================>.........] - ETA: 0s - loss: 1.2914 - accuracy: 0.5477 1162/1563 [=====================>........] - ETA: 0s - loss: 1.2925 - accuracy: 0.5474 1200/1563 [======================>.......] - ETA: 0s - loss: 1.2907 - accuracy: 0.5481 1237/1563 [======================>.......] - ETA: 0s - loss: 1.2894 - accuracy: 0.5487 1275/1563 [=======================>......] - ETA: 0s - loss: 1.2905 - accuracy: 0.5479 1313/1563 [========================>.....] - ETA: 0s - loss: 1.2885 - accuracy: 0.5484 1351/1563 [========================>.....] - ETA: 0s - loss: 1.2885 - accuracy: 0.5484 1389/1563 [=========================>....] - ETA: 0s - loss: 1.2876 - accuracy: 0.5485 1426/1563 [==========================>...] - ETA: 0s - loss: 1.2871 - accuracy: 0.5488 1463/1563 [===========================>..] - ETA: 0s - loss: 1.2854 - accuracy: 0.5496 1500/1563 [===========================>..] - ETA: 0s - loss: 1.2848 - accuracy: 0.5495 1537/1563 [============================>.] - ETA: 0s - loss: 1.2830 - accuracy: 0.5500 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2829 - accuracy: 0.5499 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.2276 - accuracy: 0.6562 39/1563 [..............................] - ETA: 2s - loss: 1.2115 - accuracy: 0.5713 76/1563 [>.............................] - ETA: 2s - loss: 1.2436 - accuracy: 0.5592 114/1563 [=>............................] - ETA: 1s - loss: 1.2446 - accuracy: 0.5663 152/1563 [=>............................] - ETA: 1s - loss: 1.2395 - accuracy: 0.5711 189/1563 [==>...........................] - ETA: 1s - loss: 1.2524 - accuracy: 0.5643 226/1563 [===>..........................] - ETA: 1s - loss: 1.2439 - accuracy: 0.5675 263/1563 [====>.........................] - ETA: 1s - loss: 1.2440 - accuracy: 0.5667 300/1563 [====>.........................] - ETA: 1s - loss: 1.2404 - accuracy: 0.5696 338/1563 [=====>........................] - ETA: 1s - loss: 1.2401 - accuracy: 0.5698 376/1563 [======>.......................] - ETA: 1s - loss: 1.2437 - accuracy: 0.5691 414/1563 [======>.......................] - ETA: 1s - loss: 1.2444 - accuracy: 0.5693 452/1563 [=======>......................] - ETA: 1s - loss: 1.2429 - accuracy: 0.5692 489/1563 [========>.....................] - ETA: 1s - loss: 1.2368 - accuracy: 0.5702 527/1563 [=========>....................] - ETA: 1s - loss: 1.2391 - accuracy: 0.5692 565/1563 [=========>....................] - ETA: 1s - loss: 1.2431 - accuracy: 0.5674 603/1563 [==========>...................] - ETA: 1s - loss: 1.2466 - accuracy: 0.5648 641/1563 [===========>..................] - ETA: 1s - loss: 1.2450 - accuracy: 0.5652 679/1563 [============>.................] - ETA: 1s - loss: 1.2457 - accuracy: 0.5648 717/1563 [============>.................] - ETA: 1s - loss: 1.2475 - accuracy: 0.5638 755/1563 [=============>................] - ETA: 1s - loss: 1.2494 - accuracy: 0.5644 793/1563 [==============>...............] - ETA: 1s - loss: 1.2496 - accuracy: 0.5640 831/1563 [==============>...............] - ETA: 0s - loss: 1.2504 - accuracy: 0.5639 869/1563 [===============>..............] - ETA: 0s - loss: 1.2510 - accuracy: 0.5633 906/1563 [================>.............] - ETA: 0s - loss: 1.2499 - accuracy: 0.5631 944/1563 [=================>............] - ETA: 0s - loss: 1.2499 - accuracy: 0.5636 981/1563 [=================>............] - ETA: 0s - loss: 1.2505 - accuracy: 0.5631 1018/1563 [==================>...........] - ETA: 0s - loss: 1.2514 - accuracy: 0.5630 1055/1563 [===================>..........] - ETA: 0s - loss: 1.2507 - accuracy: 0.5627 1093/1563 [===================>..........] - ETA: 0s - loss: 1.2492 - accuracy: 0.5632 1131/1563 [====================>.........] - ETA: 0s - loss: 1.2499 - accuracy: 0.5631 1169/1563 [=====================>........] - ETA: 0s - loss: 1.2498 - accuracy: 0.5630 1207/1563 [======================>.......] - ETA: 0s - loss: 1.2509 - accuracy: 0.5626 1245/1563 [======================>.......] - ETA: 0s - loss: 1.2529 - accuracy: 0.5617 1283/1563 [=======================>......] - ETA: 0s - loss: 1.2528 - accuracy: 0.5620 1320/1563 [========================>.....] - ETA: 0s - loss: 1.2524 - accuracy: 0.5619 1357/1563 [=========================>....] - ETA: 0s - loss: 1.2527 - accuracy: 0.5619 1394/1563 [=========================>....] - ETA: 0s - loss: 1.2526 - accuracy: 0.5618 1432/1563 [==========================>...] - ETA: 0s - loss: 1.2538 - accuracy: 0.5613 1470/1563 [===========================>..] - ETA: 0s - loss: 1.2531 - accuracy: 0.5615 1508/1563 [===========================>..] - ETA: 0s - loss: 1.2522 - accuracy: 0.5613 1545/1563 [============================>.] - ETA: 0s - loss: 1.2523 - accuracy: 0.5616 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2519 - accuracy: 0.5618 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 27.594902276992798 hydro05 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins16519683384971363363.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS