Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 84649 queued and waiting for resources srun: job 84649 has been allocated resources 2024-10-12 07:43:08.046640: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-12 07:43:34.117327: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-10-12 07:44:10.165651: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-10-12 07:44:10.166226: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-10-12 07:44:10.166235: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-10-12 07:45:07.534376: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-12 07:45:10.456337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79030 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-10-12 07:45:19.415345: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:51:26 - loss: 2.3124 - accuracy: 0.1250 37/1563 [..............................] - ETA: 2s - loss: 2.2395 - accuracy: 0.1715  75/1563 [>.............................] - ETA: 2s - loss: 2.1598 - accuracy: 0.2025 113/1563 [=>............................] - ETA: 1s - loss: 2.1118 - accuracy: 0.2270 151/1563 [=>............................] - ETA: 1s - loss: 2.0792 - accuracy: 0.2432 189/1563 [==>...........................] - ETA: 1s - loss: 2.0502 - accuracy: 0.2561 227/1563 [===>..........................] - ETA: 1s - loss: 2.0255 - accuracy: 0.2642 265/1563 [====>.........................] - ETA: 1s - loss: 2.0097 - accuracy: 0.2691 303/1563 [====>.........................] - ETA: 1s - loss: 1.9946 - accuracy: 0.2745 341/1563 [=====>........................] - ETA: 1s - loss: 1.9788 - accuracy: 0.2810 379/1563 [======>.......................] - ETA: 1s - loss: 1.9701 - accuracy: 0.2864 417/1563 [=======>......................] - ETA: 1s - loss: 1.9579 - accuracy: 0.2922 455/1563 [=======>......................] - ETA: 1s - loss: 1.9482 - accuracy: 0.2966 493/1563 [========>.....................] - ETA: 1s - loss: 1.9374 - accuracy: 0.3009 531/1563 [=========>....................] - ETA: 1s - loss: 1.9319 - accuracy: 0.3040 569/1563 [=========>....................] - ETA: 1s - loss: 1.9254 - accuracy: 0.3072 607/1563 [==========>...................] - ETA: 1s - loss: 1.9184 - accuracy: 0.3103 646/1563 [===========>..................] - ETA: 1s - loss: 1.9132 - accuracy: 0.3136 684/1563 [============>.................] - ETA: 1s - loss: 1.9056 - accuracy: 0.3167 722/1563 [============>.................] - ETA: 1s - loss: 1.8998 - accuracy: 0.3195 760/1563 [=============>................] - ETA: 1s - loss: 1.8945 - accuracy: 0.3215 798/1563 [==============>...............] - ETA: 1s - loss: 1.8895 - accuracy: 0.3230 837/1563 [===============>..............] - ETA: 0s - loss: 1.8831 - accuracy: 0.3251 875/1563 [===============>..............] - ETA: 0s - loss: 1.8780 - accuracy: 0.3278 913/1563 [================>.............] - ETA: 0s - loss: 1.8744 - accuracy: 0.3298 951/1563 [=================>............] - ETA: 0s - loss: 1.8709 - accuracy: 0.3312 989/1563 [=================>............] - ETA: 0s - loss: 1.8678 - accuracy: 0.3323 1027/1563 [==================>...........] - ETA: 0s - loss: 1.8619 - accuracy: 0.3351 1065/1563 [===================>..........] - ETA: 0s - loss: 1.8573 - accuracy: 0.3374 1103/1563 [====================>.........] - ETA: 0s - loss: 1.8524 - accuracy: 0.3388 1141/1563 [====================>.........] - ETA: 0s - loss: 1.8494 - accuracy: 0.3397 1180/1563 [=====================>........] - ETA: 0s - loss: 1.8466 - accuracy: 0.3407 1218/1563 [======================>.......] - ETA: 0s - loss: 1.8424 - accuracy: 0.3426 1257/1563 [=======================>......] - ETA: 0s - loss: 1.8373 - accuracy: 0.3447 1295/1563 [=======================>......] - ETA: 0s - loss: 1.8319 - accuracy: 0.3471 1334/1563 [========================>.....] - ETA: 0s - loss: 1.8287 - accuracy: 0.3485 1372/1563 [=========================>....] - ETA: 0s - loss: 1.8264 - accuracy: 0.3496 1411/1563 [==========================>...] - ETA: 0s - loss: 1.8236 - accuracy: 0.3509 1449/1563 [==========================>...] - ETA: 0s - loss: 1.8204 - accuracy: 0.3520 1488/1563 [===========================>..] - ETA: 0s - loss: 1.8171 - accuracy: 0.3531 1527/1563 [============================>.] - ETA: 0s - loss: 1.8136 - accuracy: 0.3542 1563/1563 [==============================] - 6s 1ms/step - loss: 1.8106 - accuracy: 0.3556 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.6469 - accuracy: 0.4688 39/1563 [..............................] - ETA: 2s - loss: 1.6997 - accuracy: 0.3982 77/1563 [>.............................] - ETA: 1s - loss: 1.6920 - accuracy: 0.4042 115/1563 [=>............................] - ETA: 1s - loss: 1.6817 - accuracy: 0.4035 153/1563 [=>............................] - ETA: 1s - loss: 1.6854 - accuracy: 0.4046 191/1563 [==>...........................] - ETA: 1s - loss: 1.6756 - accuracy: 0.4090 229/1563 [===>..........................] - ETA: 1s - loss: 1.6758 - accuracy: 0.4084 268/1563 [====>.........................] - ETA: 1s - loss: 1.6738 - accuracy: 0.4072 306/1563 [====>.........................] - ETA: 1s - loss: 1.6686 - accuracy: 0.4117 345/1563 [=====>........................] - ETA: 1s - loss: 1.6662 - accuracy: 0.4125 384/1563 [======>.......................] - ETA: 1s - loss: 1.6693 - accuracy: 0.4101 422/1563 [=======>......................] - ETA: 1s - loss: 1.6637 - accuracy: 0.4108 461/1563 [=======>......................] - ETA: 1s - loss: 1.6613 - accuracy: 0.4124 499/1563 [========>.....................] - ETA: 1s - loss: 1.6556 - accuracy: 0.4144 537/1563 [=========>....................] - ETA: 1s - loss: 1.6519 - accuracy: 0.4164 575/1563 [==========>...................] - ETA: 1s - loss: 1.6515 - accuracy: 0.4153 614/1563 [==========>...................] - ETA: 1s - loss: 1.6511 - accuracy: 0.4159 653/1563 [===========>..................] - ETA: 1s - loss: 1.6511 - accuracy: 0.4152 692/1563 [============>.................] - ETA: 1s - loss: 1.6496 - accuracy: 0.4174 730/1563 [=============>................] - ETA: 1s - loss: 1.6492 - accuracy: 0.4172 768/1563 [=============>................] - ETA: 1s - loss: 1.6470 - accuracy: 0.4183 806/1563 [==============>...............] - ETA: 0s - loss: 1.6470 - accuracy: 0.4187 844/1563 [===============>..............] - ETA: 0s - loss: 1.6440 - accuracy: 0.4198 882/1563 [===============>..............] - ETA: 0s - loss: 1.6427 - accuracy: 0.4200 920/1563 [================>.............] - ETA: 0s - loss: 1.6384 - accuracy: 0.4216 958/1563 [=================>............] - ETA: 0s - loss: 1.6387 - accuracy: 0.4210 996/1563 [==================>...........] - ETA: 0s - loss: 1.6371 - accuracy: 0.4210 1034/1563 [==================>...........] - ETA: 0s - loss: 1.6361 - accuracy: 0.4221 1072/1563 [===================>..........] - ETA: 0s - loss: 1.6350 - accuracy: 0.4225 1110/1563 [====================>.........] - ETA: 0s - loss: 1.6321 - accuracy: 0.4235 1149/1563 [=====================>........] - ETA: 0s - loss: 1.6312 - accuracy: 0.4238 1188/1563 [=====================>........] - ETA: 0s - loss: 1.6274 - accuracy: 0.4249 1226/1563 [======================>.......] - ETA: 0s - loss: 1.6271 - accuracy: 0.4252 1264/1563 [=======================>......] - ETA: 0s - loss: 1.6274 - accuracy: 0.4251 1302/1563 [=======================>......] - ETA: 0s - loss: 1.6272 - accuracy: 0.4249 1340/1563 [========================>.....] - ETA: 0s - loss: 1.6256 - accuracy: 0.4257 1378/1563 [=========================>....] - ETA: 0s - loss: 1.6254 - accuracy: 0.4261 1416/1563 [==========================>...] - ETA: 0s - loss: 1.6241 - accuracy: 0.4262 1454/1563 [==========================>...] - ETA: 0s - loss: 1.6230 - accuracy: 0.4266 1492/1563 [===========================>..] - ETA: 0s - loss: 1.6229 - accuracy: 0.4267 1530/1563 [============================>.] - ETA: 0s - loss: 1.6226 - accuracy: 0.4267 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6213 - accuracy: 0.4269 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.3695 - accuracy: 0.5000 39/1563 [..............................] - ETA: 2s - loss: 1.5389 - accuracy: 0.4559 78/1563 [>.............................] - ETA: 1s - loss: 1.5578 - accuracy: 0.4547 116/1563 [=>............................] - ETA: 1s - loss: 1.5639 - accuracy: 0.4494 154/1563 [=>............................] - ETA: 1s - loss: 1.5660 - accuracy: 0.4454 193/1563 [==>...........................] - ETA: 1s - loss: 1.5569 - accuracy: 0.4519 232/1563 [===>..........................] - ETA: 1s - loss: 1.5576 - accuracy: 0.4500 270/1563 [====>.........................] - ETA: 1s - loss: 1.5536 - accuracy: 0.4505 308/1563 [====>.........................] - ETA: 1s - loss: 1.5541 - accuracy: 0.4491 346/1563 [=====>........................] - ETA: 1s - loss: 1.5502 - accuracy: 0.4515 384/1563 [======>.......................] - ETA: 1s - loss: 1.5478 - accuracy: 0.4514 422/1563 [=======>......................] - ETA: 1s - loss: 1.5482 - accuracy: 0.4516 461/1563 [=======>......................] - ETA: 1s - loss: 1.5481 - accuracy: 0.4521 499/1563 [========>.....................] - ETA: 1s - loss: 1.5467 - accuracy: 0.4543 538/1563 [=========>....................] - ETA: 1s - loss: 1.5459 - accuracy: 0.4546 576/1563 [==========>...................] - ETA: 1s - loss: 1.5436 - accuracy: 0.4555 614/1563 [==========>...................] - ETA: 1s - loss: 1.5428 - accuracy: 0.4545 652/1563 [===========>..................] - ETA: 1s - loss: 1.5452 - accuracy: 0.4535 690/1563 [============>.................] - ETA: 1s - loss: 1.5437 - accuracy: 0.4542 728/1563 [============>.................] - ETA: 1s - loss: 1.5451 - accuracy: 0.4543 766/1563 [=============>................] - ETA: 1s - loss: 1.5438 - accuracy: 0.4552 805/1563 [==============>...............] - ETA: 1s - loss: 1.5440 - accuracy: 0.4555 844/1563 [===============>..............] - ETA: 0s - loss: 1.5412 - accuracy: 0.4570 883/1563 [===============>..............] - ETA: 0s - loss: 1.5400 - accuracy: 0.4580 921/1563 [================>.............] - ETA: 0s - loss: 1.5402 - accuracy: 0.4578 959/1563 [=================>............] - ETA: 0s - loss: 1.5382 - accuracy: 0.4583 997/1563 [==================>...........] - ETA: 0s - loss: 1.5378 - accuracy: 0.4588 1035/1563 [==================>...........] - ETA: 0s - loss: 1.5372 - accuracy: 0.4591 1073/1563 [===================>..........] - ETA: 0s - loss: 1.5378 - accuracy: 0.4588 1111/1563 [====================>.........] - ETA: 0s - loss: 1.5391 - accuracy: 0.4583 1150/1563 [=====================>........] - ETA: 0s - loss: 1.5388 - accuracy: 0.4585 1189/1563 [=====================>........] - ETA: 0s - loss: 1.5374 - accuracy: 0.4590 1228/1563 [======================>.......] - ETA: 0s - loss: 1.5363 - accuracy: 0.4594 1266/1563 [=======================>......] - ETA: 0s - loss: 1.5359 - accuracy: 0.4598 1304/1563 [========================>.....] - ETA: 0s - loss: 1.5361 - accuracy: 0.4597 1342/1563 [========================>.....] - ETA: 0s - loss: 1.5363 - accuracy: 0.4598 1380/1563 [=========================>....] - ETA: 0s - loss: 1.5360 - accuracy: 0.4599 1419/1563 [==========================>...] - ETA: 0s - loss: 1.5359 - accuracy: 0.4597 1458/1563 [==========================>...] - ETA: 0s - loss: 1.5367 - accuracy: 0.4591 1496/1563 [===========================>..] - ETA: 0s - loss: 1.5365 - accuracy: 0.4590 1534/1563 [============================>.] - ETA: 0s - loss: 1.5385 - accuracy: 0.4586 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5384 - accuracy: 0.4585 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.4855 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.4662 - accuracy: 0.4840 77/1563 [>.............................] - ETA: 1s - loss: 1.4795 - accuracy: 0.4821 115/1563 [=>............................] - ETA: 1s - loss: 1.4707 - accuracy: 0.4848 154/1563 [=>............................] - ETA: 1s - loss: 1.4679 - accuracy: 0.4834 193/1563 [==>...........................] - ETA: 1s - loss: 1.4803 - accuracy: 0.4785 232/1563 [===>..........................] - ETA: 1s - loss: 1.4857 - accuracy: 0.4752 271/1563 [====>.........................] - ETA: 1s - loss: 1.4855 - accuracy: 0.4732 310/1563 [====>.........................] - ETA: 1s - loss: 1.4873 - accuracy: 0.4745 349/1563 [=====>........................] - ETA: 1s - loss: 1.4869 - accuracy: 0.4765 387/1563 [======>.......................] - ETA: 1s - loss: 1.4857 - accuracy: 0.4788 425/1563 [=======>......................] - ETA: 1s - loss: 1.4813 - accuracy: 0.4795 464/1563 [=======>......................] - ETA: 1s - loss: 1.4818 - accuracy: 0.4803 502/1563 [========>.....................] - ETA: 1s - loss: 1.4836 - accuracy: 0.4800 541/1563 [=========>....................] - ETA: 1s - loss: 1.4862 - accuracy: 0.4787 579/1563 [==========>...................] - ETA: 1s - loss: 1.4853 - accuracy: 0.4792 617/1563 [==========>...................] - ETA: 1s - loss: 1.4854 - accuracy: 0.4796 655/1563 [===========>..................] - ETA: 1s - loss: 1.4854 - accuracy: 0.4789 693/1563 [============>.................] - ETA: 1s - loss: 1.4862 - accuracy: 0.4792 731/1563 [=============>................] - ETA: 1s - loss: 1.4867 - accuracy: 0.4789 769/1563 [=============>................] - ETA: 1s - loss: 1.4871 - accuracy: 0.4790 808/1563 [==============>...............] - ETA: 0s - loss: 1.4874 - accuracy: 0.4788 847/1563 [===============>..............] - ETA: 0s - loss: 1.4878 - accuracy: 0.4782 885/1563 [===============>..............] - ETA: 0s - loss: 1.4849 - accuracy: 0.4797 923/1563 [================>.............] - ETA: 0s - loss: 1.4845 - accuracy: 0.4801 961/1563 [=================>............] - ETA: 0s - loss: 1.4831 - accuracy: 0.4805 999/1563 [==================>...........] - ETA: 0s - loss: 1.4840 - accuracy: 0.4797 1037/1563 [==================>...........] - ETA: 0s - loss: 1.4835 - accuracy: 0.4793 1076/1563 [===================>..........] - ETA: 0s - loss: 1.4835 - accuracy: 0.4788 1114/1563 [====================>.........] - ETA: 0s - loss: 1.4834 - accuracy: 0.4786 1153/1563 [=====================>........] - ETA: 0s - loss: 1.4826 - accuracy: 0.4787 1191/1563 [=====================>........] - ETA: 0s - loss: 1.4827 - accuracy: 0.4782 1229/1563 [======================>.......] - ETA: 0s - loss: 1.4822 - accuracy: 0.4786 1267/1563 [=======================>......] - ETA: 0s - loss: 1.4821 - accuracy: 0.4788 1305/1563 [========================>.....] - ETA: 0s - loss: 1.4826 - accuracy: 0.4786 1343/1563 [========================>.....] - ETA: 0s - loss: 1.4826 - accuracy: 0.4785 1382/1563 [=========================>....] - ETA: 0s - loss: 1.4815 - accuracy: 0.4789 1421/1563 [==========================>...] - ETA: 0s - loss: 1.4806 - accuracy: 0.4788 1460/1563 [===========================>..] - ETA: 0s - loss: 1.4809 - accuracy: 0.4789 1499/1563 [===========================>..] - ETA: 0s - loss: 1.4800 - accuracy: 0.4788 1538/1563 [============================>.] - ETA: 0s - loss: 1.4799 - accuracy: 0.4788 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4796 - accuracy: 0.4789 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.5506 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.4903 - accuracy: 0.4607 78/1563 [>.............................] - ETA: 1s - loss: 1.4711 - accuracy: 0.4800 117/1563 [=>............................] - ETA: 1s - loss: 1.4557 - accuracy: 0.4912 155/1563 [=>............................] - ETA: 1s - loss: 1.4451 - accuracy: 0.4962 194/1563 [==>...........................] - ETA: 1s - loss: 1.4357 - accuracy: 0.5006 232/1563 [===>..........................] - ETA: 1s - loss: 1.4297 - accuracy: 0.5008 271/1563 [====>.........................] - ETA: 1s - loss: 1.4302 - accuracy: 0.4985 309/1563 [====>.........................] - ETA: 1s - loss: 1.4373 - accuracy: 0.4964 347/1563 [=====>........................] - ETA: 1s - loss: 1.4343 - accuracy: 0.4959 385/1563 [======>.......................] - ETA: 1s - loss: 1.4339 - accuracy: 0.4974 424/1563 [=======>......................] - ETA: 1s - loss: 1.4365 - accuracy: 0.4946 463/1563 [=======>......................] - ETA: 1s - loss: 1.4320 - accuracy: 0.4968 501/1563 [========>.....................] - ETA: 1s - loss: 1.4378 - accuracy: 0.4935 539/1563 [=========>....................] - ETA: 1s - loss: 1.4379 - accuracy: 0.4928 577/1563 [==========>...................] - ETA: 1s - loss: 1.4355 - accuracy: 0.4945 616/1563 [==========>...................] - ETA: 1s - loss: 1.4372 - accuracy: 0.4930 654/1563 [===========>..................] - ETA: 1s - loss: 1.4337 - accuracy: 0.4945 693/1563 [============>.................] - ETA: 1s - loss: 1.4322 - accuracy: 0.4946 732/1563 [=============>................] - ETA: 1s - loss: 1.4335 - accuracy: 0.4941 770/1563 [=============>................] - ETA: 1s - loss: 1.4333 - accuracy: 0.4942 809/1563 [==============>...............] - ETA: 0s - loss: 1.4315 - accuracy: 0.4949 847/1563 [===============>..............] - ETA: 0s - loss: 1.4329 - accuracy: 0.4943 885/1563 [===============>..............] - ETA: 0s - loss: 1.4327 - accuracy: 0.4946 923/1563 [================>.............] - ETA: 0s - loss: 1.4324 - accuracy: 0.4948 961/1563 [=================>............] - ETA: 0s - loss: 1.4312 - accuracy: 0.4959 999/1563 [==================>...........] - ETA: 0s - loss: 1.4303 - accuracy: 0.4966 1037/1563 [==================>...........] - ETA: 0s - loss: 1.4298 - accuracy: 0.4966 1075/1563 [===================>..........] - ETA: 0s - loss: 1.4311 - accuracy: 0.4961 1114/1563 [====================>.........] - ETA: 0s - loss: 1.4332 - accuracy: 0.4952 1152/1563 [=====================>........] - ETA: 0s - loss: 1.4339 - accuracy: 0.4951 1190/1563 [=====================>........] - ETA: 0s - loss: 1.4346 - accuracy: 0.4949 1228/1563 [======================>.......] - ETA: 0s - loss: 1.4333 - accuracy: 0.4957 1266/1563 [=======================>......] - ETA: 0s - loss: 1.4343 - accuracy: 0.4953 1304/1563 [========================>.....] - ETA: 0s - loss: 1.4339 - accuracy: 0.4959 1342/1563 [========================>.....] - ETA: 0s - loss: 1.4319 - accuracy: 0.4965 1380/1563 [=========================>....] - ETA: 0s - loss: 1.4313 - accuracy: 0.4969 1418/1563 [==========================>...] - ETA: 0s - loss: 1.4317 - accuracy: 0.4962 1457/1563 [==========================>...] - ETA: 0s - loss: 1.4311 - accuracy: 0.4963 1496/1563 [===========================>..] - ETA: 0s - loss: 1.4305 - accuracy: 0.4967 1534/1563 [============================>.] - ETA: 0s - loss: 1.4309 - accuracy: 0.4966 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4313 - accuracy: 0.4964 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 0.9298 - accuracy: 0.6875 40/1563 [..............................] - ETA: 2s - loss: 1.3821 - accuracy: 0.5172 78/1563 [>.............................] - ETA: 1s - loss: 1.3752 - accuracy: 0.5152 117/1563 [=>............................] - ETA: 1s - loss: 1.3701 - accuracy: 0.5256 155/1563 [=>............................] - ETA: 1s - loss: 1.3733 - accuracy: 0.5220 193/1563 [==>...........................] - ETA: 1s - loss: 1.3725 - accuracy: 0.5209 232/1563 [===>..........................] - ETA: 1s - loss: 1.3767 - accuracy: 0.5195 271/1563 [====>.........................] - ETA: 1s - loss: 1.3743 - accuracy: 0.5198 310/1563 [====>.........................] - ETA: 1s - loss: 1.3783 - accuracy: 0.5172 348/1563 [=====>........................] - ETA: 1s - loss: 1.3755 - accuracy: 0.5189 386/1563 [======>.......................] - ETA: 1s - loss: 1.3725 - accuracy: 0.5195 424/1563 [=======>......................] - ETA: 1s - loss: 1.3745 - accuracy: 0.5193 462/1563 [=======>......................] - ETA: 1s - loss: 1.3758 - accuracy: 0.5189 501/1563 [========>.....................] - ETA: 1s - loss: 1.3751 - accuracy: 0.5190 540/1563 [=========>....................] - ETA: 1s - loss: 1.3785 - accuracy: 0.5174 579/1563 [==========>...................] - ETA: 1s - loss: 1.3782 - accuracy: 0.5179 617/1563 [==========>...................] - ETA: 1s - loss: 1.3795 - accuracy: 0.5167 655/1563 [===========>..................] - ETA: 1s - loss: 1.3796 - accuracy: 0.5169 694/1563 [============>.................] - ETA: 1s - loss: 1.3833 - accuracy: 0.5164 732/1563 [=============>................] - ETA: 1s - loss: 1.3824 - accuracy: 0.5166 771/1563 [=============>................] - ETA: 1s - loss: 1.3848 - accuracy: 0.5160 810/1563 [==============>...............] - ETA: 0s - loss: 1.3848 - accuracy: 0.5155 849/1563 [===============>..............] - ETA: 0s - loss: 1.3850 - accuracy: 0.5155 888/1563 [================>.............] - ETA: 0s - loss: 1.3840 - accuracy: 0.5162 926/1563 [================>.............] - ETA: 0s - loss: 1.3842 - accuracy: 0.5159 964/1563 [=================>............] - ETA: 0s - loss: 1.3852 - accuracy: 0.5158 1002/1563 [==================>...........] - ETA: 0s - loss: 1.3856 - accuracy: 0.5157 1040/1563 [==================>...........] - ETA: 0s - loss: 1.3860 - accuracy: 0.5155 1078/1563 [===================>..........] - ETA: 0s - loss: 1.3867 - accuracy: 0.5152 1117/1563 [====================>.........] - ETA: 0s - loss: 1.3890 - accuracy: 0.5145 1156/1563 [=====================>........] - ETA: 0s - loss: 1.3898 - accuracy: 0.5141 1194/1563 [=====================>........] - ETA: 0s - loss: 1.3905 - accuracy: 0.5137 1232/1563 [======================>.......] - ETA: 0s - loss: 1.3905 - accuracy: 0.5134 1270/1563 [=======================>......] - ETA: 0s - loss: 1.3892 - accuracy: 0.5132 1308/1563 [========================>.....] - ETA: 0s - loss: 1.3896 - accuracy: 0.5132 1346/1563 [========================>.....] - ETA: 0s - loss: 1.3901 - accuracy: 0.5133 1385/1563 [=========================>....] - ETA: 0s - loss: 1.3900 - accuracy: 0.5139 1424/1563 [==========================>...] - ETA: 0s - loss: 1.3893 - accuracy: 0.5141 1462/1563 [===========================>..] - ETA: 0s - loss: 1.3896 - accuracy: 0.5140 1501/1563 [===========================>..] - ETA: 0s - loss: 1.3892 - accuracy: 0.5143 1539/1563 [============================>.] - ETA: 0s - loss: 1.3888 - accuracy: 0.5144 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3888 - accuracy: 0.5139 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.5200 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.3828 - accuracy: 0.5168 77/1563 [>.............................] - ETA: 1s - loss: 1.3695 - accuracy: 0.5191 115/1563 [=>............................] - ETA: 1s - loss: 1.3590 - accuracy: 0.5234 154/1563 [=>............................] - ETA: 1s - loss: 1.3603 - accuracy: 0.5229 193/1563 [==>...........................] - ETA: 1s - loss: 1.3577 - accuracy: 0.5261 231/1563 [===>..........................] - ETA: 1s - loss: 1.3558 - accuracy: 0.5267 269/1563 [====>.........................] - ETA: 1s - loss: 1.3555 - accuracy: 0.5272 307/1563 [====>.........................] - ETA: 1s - loss: 1.3536 - accuracy: 0.5293 346/1563 [=====>........................] - ETA: 1s - loss: 1.3546 - accuracy: 0.5269 384/1563 [======>.......................] - ETA: 1s - loss: 1.3519 - accuracy: 0.5269 422/1563 [=======>......................] - ETA: 1s - loss: 1.3511 - accuracy: 0.5266 461/1563 [=======>......................] - ETA: 1s - loss: 1.3511 - accuracy: 0.5275 500/1563 [========>.....................] - ETA: 1s - loss: 1.3525 - accuracy: 0.5264 538/1563 [=========>....................] - ETA: 1s - loss: 1.3496 - accuracy: 0.5281 576/1563 [==========>...................] - ETA: 1s - loss: 1.3506 - accuracy: 0.5279 614/1563 [==========>...................] - ETA: 1s - loss: 1.3521 - accuracy: 0.5267 652/1563 [===========>..................] - ETA: 1s - loss: 1.3506 - accuracy: 0.5269 690/1563 [============>.................] - ETA: 1s - loss: 1.3517 - accuracy: 0.5269 728/1563 [============>.................] - ETA: 1s - loss: 1.3499 - accuracy: 0.5276 767/1563 [=============>................] - ETA: 1s - loss: 1.3499 - accuracy: 0.5262 805/1563 [==============>...............] - ETA: 0s - loss: 1.3506 - accuracy: 0.5264 844/1563 [===============>..............] - ETA: 0s - loss: 1.3494 - accuracy: 0.5268 882/1563 [===============>..............] - ETA: 0s - loss: 1.3487 - accuracy: 0.5265 920/1563 [================>.............] - ETA: 0s - loss: 1.3481 - accuracy: 0.5268 958/1563 [=================>............] - ETA: 0s - loss: 1.3493 - accuracy: 0.5270 996/1563 [==================>...........] - ETA: 0s - loss: 1.3490 - accuracy: 0.5265 1035/1563 [==================>...........] - ETA: 0s - loss: 1.3481 - accuracy: 0.5271 1074/1563 [===================>..........] - ETA: 0s - loss: 1.3487 - accuracy: 0.5264 1113/1563 [====================>.........] - ETA: 0s - loss: 1.3481 - accuracy: 0.5261 1151/1563 [=====================>........] - ETA: 0s - loss: 1.3495 - accuracy: 0.5258 1189/1563 [=====================>........] - ETA: 0s - loss: 1.3505 - accuracy: 0.5255 1227/1563 [======================>.......] - ETA: 0s - loss: 1.3503 - accuracy: 0.5261 1265/1563 [=======================>......] - ETA: 0s - loss: 1.3504 - accuracy: 0.5257 1303/1563 [========================>.....] - ETA: 0s - loss: 1.3503 - accuracy: 0.5260 1342/1563 [========================>.....] - ETA: 0s - loss: 1.3506 - accuracy: 0.5258 1381/1563 [=========================>....] - ETA: 0s - loss: 1.3499 - accuracy: 0.5258 1420/1563 [==========================>...] - ETA: 0s - loss: 1.3493 - accuracy: 0.5261 1458/1563 [==========================>...] - ETA: 0s - loss: 1.3497 - accuracy: 0.5255 1496/1563 [===========================>..] - ETA: 0s - loss: 1.3495 - accuracy: 0.5256 1534/1563 [============================>.] - ETA: 0s - loss: 1.3495 - accuracy: 0.5259 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3492 - accuracy: 0.5261 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.1514 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.3132 - accuracy: 0.5361 78/1563 [>.............................] - ETA: 1s - loss: 1.3220 - accuracy: 0.5361 117/1563 [=>............................] - ETA: 1s - loss: 1.3234 - accuracy: 0.5304 156/1563 [=>............................] - ETA: 1s - loss: 1.3221 - accuracy: 0.5387 195/1563 [==>...........................] - ETA: 1s - loss: 1.3144 - accuracy: 0.5399 233/1563 [===>..........................] - ETA: 1s - loss: 1.3122 - accuracy: 0.5409 271/1563 [====>.........................] - ETA: 1s - loss: 1.3119 - accuracy: 0.5404 310/1563 [====>.........................] - ETA: 1s - loss: 1.3135 - accuracy: 0.5397 348/1563 [=====>........................] - ETA: 1s - loss: 1.3102 - accuracy: 0.5402 386/1563 [======>.......................] - ETA: 1s - loss: 1.3098 - accuracy: 0.5413 425/1563 [=======>......................] - ETA: 1s - loss: 1.3164 - accuracy: 0.5385 463/1563 [=======>......................] - ETA: 1s - loss: 1.3176 - accuracy: 0.5372 501/1563 [========>.....................] - ETA: 1s - loss: 1.3193 - accuracy: 0.5367 539/1563 [=========>....................] - ETA: 1s - loss: 1.3183 - accuracy: 0.5365 577/1563 [==========>...................] - ETA: 1s - loss: 1.3182 - accuracy: 0.5370 615/1563 [==========>...................] - ETA: 1s - loss: 1.3167 - accuracy: 0.5377 654/1563 [===========>..................] - ETA: 1s - loss: 1.3161 - accuracy: 0.5379 692/1563 [============>.................] - ETA: 1s - loss: 1.3167 - accuracy: 0.5384 731/1563 [=============>................] - ETA: 1s - loss: 1.3193 - accuracy: 0.5370 770/1563 [=============>................] - ETA: 1s - loss: 1.3204 - accuracy: 0.5365 808/1563 [==============>...............] - ETA: 0s - loss: 1.3197 - accuracy: 0.5367 846/1563 [===============>..............] - ETA: 0s - loss: 1.3198 - accuracy: 0.5373 884/1563 [===============>..............] - ETA: 0s - loss: 1.3212 - accuracy: 0.5366 922/1563 [================>.............] - ETA: 0s - loss: 1.3220 - accuracy: 0.5361 960/1563 [=================>............] - ETA: 0s - loss: 1.3224 - accuracy: 0.5361 998/1563 [==================>...........] - ETA: 0s - loss: 1.3203 - accuracy: 0.5370 1037/1563 [==================>...........] - ETA: 0s - loss: 1.3190 - accuracy: 0.5371 1076/1563 [===================>..........] - ETA: 0s - loss: 1.3177 - accuracy: 0.5372 1115/1563 [====================>.........] - ETA: 0s - loss: 1.3182 - accuracy: 0.5363 1153/1563 [=====================>........] - ETA: 0s - loss: 1.3186 - accuracy: 0.5353 1191/1563 [=====================>........] - ETA: 0s - loss: 1.3188 - accuracy: 0.5356 1229/1563 [======================>.......] - ETA: 0s - loss: 1.3175 - accuracy: 0.5359 1267/1563 [=======================>......] - ETA: 0s - loss: 1.3187 - accuracy: 0.5355 1305/1563 [========================>.....] - ETA: 0s - loss: 1.3186 - accuracy: 0.5359 1344/1563 [========================>.....] - ETA: 0s - loss: 1.3182 - accuracy: 0.5358 1382/1563 [=========================>....] - ETA: 0s - loss: 1.3189 - accuracy: 0.5358 1421/1563 [==========================>...] - ETA: 0s - loss: 1.3188 - accuracy: 0.5359 1459/1563 [===========================>..] - ETA: 0s - loss: 1.3158 - accuracy: 0.5373 1497/1563 [===========================>..] - ETA: 0s - loss: 1.3158 - accuracy: 0.5372 1535/1563 [============================>.] - ETA: 0s - loss: 1.3155 - accuracy: 0.5373 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3150 - accuracy: 0.5373 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.5591 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.2911 - accuracy: 0.5449 77/1563 [>.............................] - ETA: 1s - loss: 1.2704 - accuracy: 0.5475 115/1563 [=>............................] - ETA: 1s - loss: 1.2635 - accuracy: 0.5538 153/1563 [=>............................] - ETA: 1s - loss: 1.2772 - accuracy: 0.5484 191/1563 [==>...........................] - ETA: 1s - loss: 1.2885 - accuracy: 0.5448 229/1563 [===>..........................] - ETA: 1s - loss: 1.2824 - accuracy: 0.5482 267/1563 [====>.........................] - ETA: 1s - loss: 1.2830 - accuracy: 0.5476 305/1563 [====>.........................] - ETA: 1s - loss: 1.2852 - accuracy: 0.5464 343/1563 [=====>........................] - ETA: 1s - loss: 1.2794 - accuracy: 0.5504 381/1563 [======>.......................] - ETA: 1s - loss: 1.2797 - accuracy: 0.5505 420/1563 [=======>......................] - ETA: 1s - loss: 1.2768 - accuracy: 0.5534 459/1563 [=======>......................] - ETA: 1s - loss: 1.2774 - accuracy: 0.5533 498/1563 [========>.....................] - ETA: 1s - loss: 1.2793 - accuracy: 0.5530 536/1563 [=========>....................] - ETA: 1s - loss: 1.2803 - accuracy: 0.5525 574/1563 [==========>...................] - ETA: 1s - loss: 1.2799 - accuracy: 0.5524 612/1563 [==========>...................] - ETA: 1s - loss: 1.2817 - accuracy: 0.5518 650/1563 [===========>..................] - ETA: 1s - loss: 1.2828 - accuracy: 0.5512 689/1563 [============>.................] - ETA: 1s - loss: 1.2836 - accuracy: 0.5507 727/1563 [============>.................] - ETA: 1s - loss: 1.2848 - accuracy: 0.5499 765/1563 [=============>................] - ETA: 1s - loss: 1.2844 - accuracy: 0.5508 804/1563 [==============>...............] - ETA: 1s - loss: 1.2848 - accuracy: 0.5505 842/1563 [===============>..............] - ETA: 0s - loss: 1.2830 - accuracy: 0.5505 880/1563 [===============>..............] - ETA: 0s - loss: 1.2813 - accuracy: 0.5522 918/1563 [================>.............] - ETA: 0s - loss: 1.2792 - accuracy: 0.5529 956/1563 [=================>............] - ETA: 0s - loss: 1.2813 - accuracy: 0.5519 994/1563 [==================>...........] - ETA: 0s - loss: 1.2809 - accuracy: 0.5521 1033/1563 [==================>...........] - ETA: 0s - loss: 1.2805 - accuracy: 0.5519 1072/1563 [===================>..........] - ETA: 0s - loss: 1.2800 - accuracy: 0.5517 1110/1563 [====================>.........] - ETA: 0s - loss: 1.2797 - accuracy: 0.5518 1148/1563 [=====================>........] - ETA: 0s - loss: 1.2799 - accuracy: 0.5519 1186/1563 [=====================>........] - ETA: 0s - loss: 1.2792 - accuracy: 0.5523 1224/1563 [======================>.......] - ETA: 0s - loss: 1.2780 - accuracy: 0.5527 1262/1563 [=======================>......] - ETA: 0s - loss: 1.2785 - accuracy: 0.5522 1300/1563 [=======================>......] - ETA: 0s - loss: 1.2797 - accuracy: 0.5520 1339/1563 [========================>.....] - ETA: 0s - loss: 1.2792 - accuracy: 0.5526 1378/1563 [=========================>....] - ETA: 0s - loss: 1.2806 - accuracy: 0.5521 1416/1563 [==========================>...] - ETA: 0s - loss: 1.2806 - accuracy: 0.5519 1454/1563 [==========================>...] - ETA: 0s - loss: 1.2806 - accuracy: 0.5518 1493/1563 [===========================>..] - ETA: 0s - loss: 1.2795 - accuracy: 0.5520 1531/1563 [============================>.] - ETA: 0s - loss: 1.2805 - accuracy: 0.5515 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2811 - accuracy: 0.5511 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.3121 - accuracy: 0.5625 40/1563 [..............................] - ETA: 2s - loss: 1.1949 - accuracy: 0.6078 78/1563 [>.............................] - ETA: 1s - loss: 1.2113 - accuracy: 0.5917 117/1563 [=>............................] - ETA: 1s - loss: 1.2169 - accuracy: 0.5772 156/1563 [=>............................] - ETA: 1s - loss: 1.2235 - accuracy: 0.5723 194/1563 [==>...........................] - ETA: 1s - loss: 1.2240 - accuracy: 0.5714 232/1563 [===>..........................] - ETA: 1s - loss: 1.2188 - accuracy: 0.5746 270/1563 [====>.........................] - ETA: 1s - loss: 1.2340 - accuracy: 0.5696 308/1563 [====>.........................] - ETA: 1s - loss: 1.2363 - accuracy: 0.5684 346/1563 [=====>........................] - ETA: 1s - loss: 1.2339 - accuracy: 0.5675 384/1563 [======>.......................] - ETA: 1s - loss: 1.2385 - accuracy: 0.5653 422/1563 [=======>......................] - ETA: 1s - loss: 1.2381 - accuracy: 0.5654 460/1563 [=======>......................] - ETA: 1s - loss: 1.2340 - accuracy: 0.5668 498/1563 [========>.....................] - ETA: 1s - loss: 1.2346 - accuracy: 0.5662 536/1563 [=========>....................] - ETA: 1s - loss: 1.2354 - accuracy: 0.5655 574/1563 [==========>...................] - ETA: 1s - loss: 1.2376 - accuracy: 0.5641 612/1563 [==========>...................] - ETA: 1s - loss: 1.2390 - accuracy: 0.5632 650/1563 [===========>..................] - ETA: 1s - loss: 1.2404 - accuracy: 0.5630 688/1563 [============>.................] - ETA: 1s - loss: 1.2399 - accuracy: 0.5643 726/1563 [============>.................] - ETA: 1s - loss: 1.2428 - accuracy: 0.5643 764/1563 [=============>................] - ETA: 1s - loss: 1.2431 - accuracy: 0.5637 803/1563 [==============>...............] - ETA: 1s - loss: 1.2442 - accuracy: 0.5629 841/1563 [===============>..............] - ETA: 0s - loss: 1.2417 - accuracy: 0.5644 879/1563 [===============>..............] - ETA: 0s - loss: 1.2449 - accuracy: 0.5637 917/1563 [================>.............] - ETA: 0s - loss: 1.2468 - accuracy: 0.5634 955/1563 [=================>............] - ETA: 0s - loss: 1.2480 - accuracy: 0.5634 993/1563 [==================>...........] - ETA: 0s - loss: 1.2486 - accuracy: 0.5628 1031/1563 [==================>...........] - ETA: 0s - loss: 1.2485 - accuracy: 0.5634 1069/1563 [===================>..........] - ETA: 0s - loss: 1.2493 - accuracy: 0.5627 1107/1563 [====================>.........] - ETA: 0s - loss: 1.2484 - accuracy: 0.5630 1145/1563 [====================>.........] - ETA: 0s - loss: 1.2480 - accuracy: 0.5625 1184/1563 [=====================>........] - ETA: 0s - loss: 1.2475 - accuracy: 0.5627 1222/1563 [======================>.......] - ETA: 0s - loss: 1.2495 - accuracy: 0.5626 1260/1563 [=======================>......] - ETA: 0s - loss: 1.2507 - accuracy: 0.5624 1298/1563 [=======================>......] - ETA: 0s - loss: 1.2509 - accuracy: 0.5623 1336/1563 [========================>.....] - ETA: 0s - loss: 1.2516 - accuracy: 0.5622 1374/1563 [=========================>....] - ETA: 0s - loss: 1.2520 - accuracy: 0.5621 1413/1563 [==========================>...] - ETA: 0s - loss: 1.2509 - accuracy: 0.5622 1452/1563 [==========================>...] - ETA: 0s - loss: 1.2495 - accuracy: 0.5628 1491/1563 [===========================>..] - ETA: 0s - loss: 1.2489 - accuracy: 0.5629 1529/1563 [============================>.] - ETA: 0s - loss: 1.2491 - accuracy: 0.5628 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2495 - accuracy: 0.5623 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 39.140345096588135 hydro05 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins18045683576075885031.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS