Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 90245 queued and waiting for resources srun: job 90245 has been allocated resources 2024-10-20 07:43:06.556321: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-20 07:43:29.604896: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-10-20 07:43:57.878335: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-10-20 07:43:57.878956: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-10-20 07:43:57.878965: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-10-20 07:44:41.256901: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-20 07:44:43.305861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-10-20 07:44:50.228025: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:42:26 - loss: 2.2525 - accuracy: 0.2500 38/1563 [..............................] - ETA: 2s - loss: 2.2410 - accuracy: 0.1711  76/1563 [>.............................] - ETA: 2s - loss: 2.1797 - accuracy: 0.1945 114/1563 [=>............................] - ETA: 1s - loss: 2.1237 - accuracy: 0.2223 153/1563 [=>............................] - ETA: 1s - loss: 2.0963 - accuracy: 0.2371 191/1563 [==>...........................] - ETA: 1s - loss: 2.0723 - accuracy: 0.2472 230/1563 [===>..........................] - ETA: 1s - loss: 2.0495 - accuracy: 0.2594 269/1563 [====>.........................] - ETA: 1s - loss: 2.0283 - accuracy: 0.2717 308/1563 [====>.........................] - ETA: 1s - loss: 2.0119 - accuracy: 0.2779 347/1563 [=====>........................] - ETA: 1s - loss: 2.0002 - accuracy: 0.2830 386/1563 [======>.......................] - ETA: 1s - loss: 1.9859 - accuracy: 0.2892 424/1563 [=======>......................] - ETA: 1s - loss: 1.9739 - accuracy: 0.2929 463/1563 [=======>......................] - ETA: 1s - loss: 1.9648 - accuracy: 0.2966 502/1563 [========>.....................] - ETA: 1s - loss: 1.9555 - accuracy: 0.3000 541/1563 [=========>....................] - ETA: 1s - loss: 1.9452 - accuracy: 0.3031 580/1563 [==========>...................] - ETA: 1s - loss: 1.9354 - accuracy: 0.3066 619/1563 [==========>...................] - ETA: 1s - loss: 1.9276 - accuracy: 0.3091 658/1563 [===========>..................] - ETA: 1s - loss: 1.9218 - accuracy: 0.3124 697/1563 [============>.................] - ETA: 1s - loss: 1.9143 - accuracy: 0.3155 736/1563 [=============>................] - ETA: 1s - loss: 1.9075 - accuracy: 0.3179 774/1563 [=============>................] - ETA: 1s - loss: 1.9012 - accuracy: 0.3206 813/1563 [==============>...............] - ETA: 0s - loss: 1.8955 - accuracy: 0.3223 852/1563 [===============>..............] - ETA: 0s - loss: 1.8901 - accuracy: 0.3246 891/1563 [================>.............] - ETA: 0s - loss: 1.8853 - accuracy: 0.3266 930/1563 [================>.............] - ETA: 0s - loss: 1.8774 - accuracy: 0.3301 969/1563 [=================>............] - ETA: 0s - loss: 1.8730 - accuracy: 0.3314 1008/1563 [==================>...........] - ETA: 0s - loss: 1.8680 - accuracy: 0.3333 1047/1563 [===================>..........] - ETA: 0s - loss: 1.8644 - accuracy: 0.3346 1085/1563 [===================>..........] - ETA: 0s - loss: 1.8593 - accuracy: 0.3360 1124/1563 [====================>.........] - ETA: 0s - loss: 1.8547 - accuracy: 0.3382 1163/1563 [=====================>........] - ETA: 0s - loss: 1.8507 - accuracy: 0.3397 1202/1563 [======================>.......] - ETA: 0s - loss: 1.8452 - accuracy: 0.3413 1241/1563 [======================>.......] - ETA: 0s - loss: 1.8417 - accuracy: 0.3431 1280/1563 [=======================>......] - ETA: 0s - loss: 1.8384 - accuracy: 0.3445 1319/1563 [========================>.....] - ETA: 0s - loss: 1.8361 - accuracy: 0.3453 1358/1563 [=========================>....] - ETA: 0s - loss: 1.8336 - accuracy: 0.3465 1397/1563 [=========================>....] - ETA: 0s - loss: 1.8299 - accuracy: 0.3480 1436/1563 [==========================>...] - ETA: 0s - loss: 1.8278 - accuracy: 0.3489 1475/1563 [===========================>..] - ETA: 0s - loss: 1.8235 - accuracy: 0.3508 1514/1563 [============================>.] - ETA: 0s - loss: 1.8203 - accuracy: 0.3526 1553/1563 [============================>.] - ETA: 0s - loss: 1.8179 - accuracy: 0.3535 1563/1563 [==============================] - 6s 1ms/step - loss: 1.8178 - accuracy: 0.3535 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.9457 - accuracy: 0.1875 40/1563 [..............................] - ETA: 2s - loss: 1.6519 - accuracy: 0.4141 79/1563 [>.............................] - ETA: 1s - loss: 1.6554 - accuracy: 0.4189 118/1563 [=>............................] - ETA: 1s - loss: 1.6414 - accuracy: 0.4240 157/1563 [==>...........................] - ETA: 1s - loss: 1.6485 - accuracy: 0.4218 196/1563 [==>...........................] - ETA: 1s - loss: 1.6557 - accuracy: 0.4217 235/1563 [===>..........................] - ETA: 1s - loss: 1.6578 - accuracy: 0.4215 274/1563 [====>.........................] - ETA: 1s - loss: 1.6629 - accuracy: 0.4174 313/1563 [=====>........................] - ETA: 1s - loss: 1.6584 - accuracy: 0.4196 352/1563 [=====>........................] - ETA: 1s - loss: 1.6578 - accuracy: 0.4193 390/1563 [======>.......................] - ETA: 1s - loss: 1.6534 - accuracy: 0.4218 429/1563 [=======>......................] - ETA: 1s - loss: 1.6485 - accuracy: 0.4234 468/1563 [=======>......................] - ETA: 1s - loss: 1.6514 - accuracy: 0.4214 507/1563 [========>.....................] - ETA: 1s - loss: 1.6515 - accuracy: 0.4223 546/1563 [=========>....................] - ETA: 1s - loss: 1.6501 - accuracy: 0.4230 585/1563 [==========>...................] - ETA: 1s - loss: 1.6471 - accuracy: 0.4239 624/1563 [==========>...................] - ETA: 1s - loss: 1.6454 - accuracy: 0.4245 663/1563 [===========>..................] - ETA: 1s - loss: 1.6458 - accuracy: 0.4231 702/1563 [============>.................] - ETA: 1s - loss: 1.6454 - accuracy: 0.4232 741/1563 [=============>................] - ETA: 1s - loss: 1.6429 - accuracy: 0.4240 780/1563 [=============>................] - ETA: 1s - loss: 1.6426 - accuracy: 0.4242 819/1563 [==============>...............] - ETA: 0s - loss: 1.6433 - accuracy: 0.4233 858/1563 [===============>..............] - ETA: 0s - loss: 1.6438 - accuracy: 0.4237 897/1563 [================>.............] - ETA: 0s - loss: 1.6423 - accuracy: 0.4238 936/1563 [================>.............] - ETA: 0s - loss: 1.6397 - accuracy: 0.4244 974/1563 [=================>............] - ETA: 0s - loss: 1.6389 - accuracy: 0.4254 1012/1563 [==================>...........] - ETA: 0s - loss: 1.6388 - accuracy: 0.4251 1051/1563 [===================>..........] - ETA: 0s - loss: 1.6377 - accuracy: 0.4253 1090/1563 [===================>..........] - ETA: 0s - loss: 1.6358 - accuracy: 0.4256 1129/1563 [====================>.........] - ETA: 0s - loss: 1.6356 - accuracy: 0.4251 1168/1563 [=====================>........] - ETA: 0s - loss: 1.6340 - accuracy: 0.4262 1207/1563 [======================>.......] - ETA: 0s - loss: 1.6333 - accuracy: 0.4257 1245/1563 [======================>.......] - ETA: 0s - loss: 1.6317 - accuracy: 0.4259 1284/1563 [=======================>......] - ETA: 0s - loss: 1.6302 - accuracy: 0.4264 1323/1563 [========================>.....] - ETA: 0s - loss: 1.6287 - accuracy: 0.4269 1361/1563 [=========================>....] - ETA: 0s - loss: 1.6277 - accuracy: 0.4271 1400/1563 [=========================>....] - ETA: 0s - loss: 1.6278 - accuracy: 0.4264 1439/1563 [==========================>...] - ETA: 0s - loss: 1.6271 - accuracy: 0.4262 1477/1563 [===========================>..] - ETA: 0s - loss: 1.6272 - accuracy: 0.4262 1516/1563 [============================>.] - ETA: 0s - loss: 1.6255 - accuracy: 0.4267 1554/1563 [============================>.] - ETA: 0s - loss: 1.6247 - accuracy: 0.4269 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6244 - accuracy: 0.4270 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.4461 - accuracy: 0.5938 40/1563 [..............................] - ETA: 1s - loss: 1.5781 - accuracy: 0.4430 79/1563 [>.............................] - ETA: 1s - loss: 1.5582 - accuracy: 0.4525 118/1563 [=>............................] - ETA: 1s - loss: 1.5689 - accuracy: 0.4523 157/1563 [==>...........................] - ETA: 1s - loss: 1.5669 - accuracy: 0.4498 196/1563 [==>...........................] - ETA: 1s - loss: 1.5602 - accuracy: 0.4526 234/1563 [===>..........................] - ETA: 1s - loss: 1.5638 - accuracy: 0.4503 272/1563 [====>.........................] - ETA: 1s - loss: 1.5590 - accuracy: 0.4497 310/1563 [====>.........................] - ETA: 1s - loss: 1.5616 - accuracy: 0.4490 348/1563 [=====>........................] - ETA: 1s - loss: 1.5633 - accuracy: 0.4494 387/1563 [======>.......................] - ETA: 1s - loss: 1.5610 - accuracy: 0.4503 426/1563 [=======>......................] - ETA: 1s - loss: 1.5597 - accuracy: 0.4514 465/1563 [=======>......................] - ETA: 1s - loss: 1.5581 - accuracy: 0.4508 504/1563 [========>.....................] - ETA: 1s - loss: 1.5578 - accuracy: 0.4501 543/1563 [=========>....................] - ETA: 1s - loss: 1.5558 - accuracy: 0.4511 582/1563 [==========>...................] - ETA: 1s - loss: 1.5543 - accuracy: 0.4516 621/1563 [==========>...................] - ETA: 1s - loss: 1.5558 - accuracy: 0.4511 660/1563 [===========>..................] - ETA: 1s - loss: 1.5510 - accuracy: 0.4535 699/1563 [============>.................] - ETA: 1s - loss: 1.5514 - accuracy: 0.4522 737/1563 [=============>................] - ETA: 1s - loss: 1.5508 - accuracy: 0.4521 776/1563 [=============>................] - ETA: 1s - loss: 1.5477 - accuracy: 0.4535 815/1563 [==============>...............] - ETA: 0s - loss: 1.5501 - accuracy: 0.4525 854/1563 [===============>..............] - ETA: 0s - loss: 1.5488 - accuracy: 0.4528 893/1563 [================>.............] - ETA: 0s - loss: 1.5492 - accuracy: 0.4522 932/1563 [================>.............] - ETA: 0s - loss: 1.5494 - accuracy: 0.4518 971/1563 [=================>............] - ETA: 0s - loss: 1.5507 - accuracy: 0.4518 1010/1563 [==================>...........] - ETA: 0s - loss: 1.5503 - accuracy: 0.4518 1049/1563 [===================>..........] - ETA: 0s - loss: 1.5505 - accuracy: 0.4517 1088/1563 [===================>..........] - ETA: 0s - loss: 1.5493 - accuracy: 0.4527 1127/1563 [====================>.........] - ETA: 0s - loss: 1.5477 - accuracy: 0.4535 1166/1563 [=====================>........] - ETA: 0s - loss: 1.5487 - accuracy: 0.4535 1205/1563 [======================>.......] - ETA: 0s - loss: 1.5478 - accuracy: 0.4539 1244/1563 [======================>.......] - ETA: 0s - loss: 1.5473 - accuracy: 0.4541 1282/1563 [=======================>......] - ETA: 0s - loss: 1.5464 - accuracy: 0.4543 1319/1563 [========================>.....] - ETA: 0s - loss: 1.5451 - accuracy: 0.4550 1357/1563 [=========================>....] - ETA: 0s - loss: 1.5444 - accuracy: 0.4554 1394/1563 [=========================>....] - ETA: 0s - loss: 1.5438 - accuracy: 0.4557 1432/1563 [==========================>...] - ETA: 0s - loss: 1.5432 - accuracy: 0.4560 1470/1563 [===========================>..] - ETA: 0s - loss: 1.5425 - accuracy: 0.4565 1508/1563 [===========================>..] - ETA: 0s - loss: 1.5428 - accuracy: 0.4564 1545/1563 [============================>.] - ETA: 0s - loss: 1.5432 - accuracy: 0.4566 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5428 - accuracy: 0.4568 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.5236 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.4631 - accuracy: 0.4784 76/1563 [>.............................] - ETA: 2s - loss: 1.4395 - accuracy: 0.4914 113/1563 [=>............................] - ETA: 1s - loss: 1.4662 - accuracy: 0.4751 151/1563 [=>............................] - ETA: 1s - loss: 1.4729 - accuracy: 0.4785 189/1563 [==>...........................] - ETA: 1s - loss: 1.4645 - accuracy: 0.4783 227/1563 [===>..........................] - ETA: 1s - loss: 1.4701 - accuracy: 0.4738 265/1563 [====>.........................] - ETA: 1s - loss: 1.4806 - accuracy: 0.4725 302/1563 [====>.........................] - ETA: 1s - loss: 1.4756 - accuracy: 0.4744 340/1563 [=====>........................] - ETA: 1s - loss: 1.4827 - accuracy: 0.4713 378/1563 [======>.......................] - ETA: 1s - loss: 1.4877 - accuracy: 0.4688 416/1563 [======>.......................] - ETA: 1s - loss: 1.4890 - accuracy: 0.4694 454/1563 [=======>......................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4684 492/1563 [========>.....................] - ETA: 1s - loss: 1.4879 - accuracy: 0.4693 530/1563 [=========>....................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4700 568/1563 [=========>....................] - ETA: 1s - loss: 1.4863 - accuracy: 0.4717 606/1563 [==========>...................] - ETA: 1s - loss: 1.4886 - accuracy: 0.4708 644/1563 [===========>..................] - ETA: 1s - loss: 1.4897 - accuracy: 0.4703 682/1563 [============>.................] - ETA: 1s - loss: 1.4898 - accuracy: 0.4703 720/1563 [============>.................] - ETA: 1s - loss: 1.4859 - accuracy: 0.4717 758/1563 [=============>................] - ETA: 1s - loss: 1.4863 - accuracy: 0.4719 796/1563 [==============>...............] - ETA: 1s - loss: 1.4857 - accuracy: 0.4728 834/1563 [===============>..............] - ETA: 0s - loss: 1.4869 - accuracy: 0.4731 873/1563 [===============>..............] - ETA: 0s - loss: 1.4866 - accuracy: 0.4732 911/1563 [================>.............] - ETA: 0s - loss: 1.4879 - accuracy: 0.4730 950/1563 [=================>............] - ETA: 0s - loss: 1.4886 - accuracy: 0.4731 988/1563 [=================>............] - ETA: 0s - loss: 1.4862 - accuracy: 0.4744 1026/1563 [==================>...........] - ETA: 0s - loss: 1.4848 - accuracy: 0.4751 1064/1563 [===================>..........] - ETA: 0s - loss: 1.4857 - accuracy: 0.4749 1102/1563 [====================>.........] - ETA: 0s - loss: 1.4847 - accuracy: 0.4753 1140/1563 [====================>.........] - ETA: 0s - loss: 1.4846 - accuracy: 0.4752 1178/1563 [=====================>........] - ETA: 0s - loss: 1.4840 - accuracy: 0.4755 1216/1563 [======================>.......] - ETA: 0s - loss: 1.4829 - accuracy: 0.4757 1253/1563 [=======================>......] - ETA: 0s - loss: 1.4804 - accuracy: 0.4767 1291/1563 [=======================>......] - ETA: 0s - loss: 1.4795 - accuracy: 0.4771 1329/1563 [========================>.....] - ETA: 0s - loss: 1.4796 - accuracy: 0.4769 1367/1563 [=========================>....] - ETA: 0s - loss: 1.4803 - accuracy: 0.4764 1405/1563 [=========================>....] - ETA: 0s - loss: 1.4798 - accuracy: 0.4764 1443/1563 [==========================>...] - ETA: 0s - loss: 1.4805 - accuracy: 0.4760 1481/1563 [===========================>..] - ETA: 0s - loss: 1.4799 - accuracy: 0.4766 1519/1563 [============================>.] - ETA: 0s - loss: 1.4796 - accuracy: 0.4767 1557/1563 [============================>.] - ETA: 0s - loss: 1.4805 - accuracy: 0.4767 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4806 - accuracy: 0.4767 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.0800 - accuracy: 0.6250 38/1563 [..............................] - ETA: 2s - loss: 1.4162 - accuracy: 0.4877 76/1563 [>.............................] - ETA: 1s - loss: 1.4661 - accuracy: 0.4844 115/1563 [=>............................] - ETA: 1s - loss: 1.4407 - accuracy: 0.4929 154/1563 [=>............................] - ETA: 1s - loss: 1.4399 - accuracy: 0.4980 193/1563 [==>...........................] - ETA: 1s - loss: 1.4362 - accuracy: 0.4971 232/1563 [===>..........................] - ETA: 1s - loss: 1.4401 - accuracy: 0.4933 271/1563 [====>.........................] - ETA: 1s - loss: 1.4390 - accuracy: 0.4938 309/1563 [====>.........................] - ETA: 1s - loss: 1.4440 - accuracy: 0.4935 348/1563 [=====>........................] - ETA: 1s - loss: 1.4471 - accuracy: 0.4922 387/1563 [======>.......................] - ETA: 1s - loss: 1.4488 - accuracy: 0.4923 426/1563 [=======>......................] - ETA: 1s - loss: 1.4480 - accuracy: 0.4913 464/1563 [=======>......................] - ETA: 1s - loss: 1.4421 - accuracy: 0.4927 502/1563 [========>.....................] - ETA: 1s - loss: 1.4418 - accuracy: 0.4928 540/1563 [=========>....................] - ETA: 1s - loss: 1.4417 - accuracy: 0.4931 578/1563 [==========>...................] - ETA: 1s - loss: 1.4433 - accuracy: 0.4925 616/1563 [==========>...................] - ETA: 1s - loss: 1.4424 - accuracy: 0.4928 655/1563 [===========>..................] - ETA: 1s - loss: 1.4414 - accuracy: 0.4932 694/1563 [============>.................] - ETA: 1s - loss: 1.4406 - accuracy: 0.4933 733/1563 [=============>................] - ETA: 1s - loss: 1.4420 - accuracy: 0.4918 772/1563 [=============>................] - ETA: 1s - loss: 1.4408 - accuracy: 0.4930 811/1563 [==============>...............] - ETA: 0s - loss: 1.4415 - accuracy: 0.4933 850/1563 [===============>..............] - ETA: 0s - loss: 1.4415 - accuracy: 0.4937 888/1563 [================>.............] - ETA: 0s - loss: 1.4431 - accuracy: 0.4924 927/1563 [================>.............] - ETA: 0s - loss: 1.4435 - accuracy: 0.4932 966/1563 [=================>............] - ETA: 0s - loss: 1.4426 - accuracy: 0.4932 1005/1563 [==================>...........] - ETA: 0s - loss: 1.4437 - accuracy: 0.4929 1044/1563 [===================>..........] - ETA: 0s - loss: 1.4436 - accuracy: 0.4932 1083/1563 [===================>..........] - ETA: 0s - loss: 1.4418 - accuracy: 0.4938 1122/1563 [====================>.........] - ETA: 0s - loss: 1.4409 - accuracy: 0.4942 1161/1563 [=====================>........] - ETA: 0s - loss: 1.4391 - accuracy: 0.4950 1200/1563 [======================>.......] - ETA: 0s - loss: 1.4377 - accuracy: 0.4955 1239/1563 [======================>.......] - ETA: 0s - loss: 1.4374 - accuracy: 0.4952 1278/1563 [=======================>......] - ETA: 0s - loss: 1.4355 - accuracy: 0.4956 1316/1563 [========================>.....] - ETA: 0s - loss: 1.4347 - accuracy: 0.4959 1355/1563 [=========================>....] - ETA: 0s - loss: 1.4331 - accuracy: 0.4966 1393/1563 [=========================>....] - ETA: 0s - loss: 1.4329 - accuracy: 0.4968 1432/1563 [==========================>...] - ETA: 0s - loss: 1.4338 - accuracy: 0.4963 1470/1563 [===========================>..] - ETA: 0s - loss: 1.4329 - accuracy: 0.4962 1508/1563 [===========================>..] - ETA: 0s - loss: 1.4318 - accuracy: 0.4963 1545/1563 [============================>.] - ETA: 0s - loss: 1.4327 - accuracy: 0.4957 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4323 - accuracy: 0.4959 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.5037 - accuracy: 0.5312 40/1563 [..............................] - ETA: 1s - loss: 1.3964 - accuracy: 0.4977 79/1563 [>.............................] - ETA: 1s - loss: 1.4034 - accuracy: 0.4996 118/1563 [=>............................] - ETA: 1s - loss: 1.4059 - accuracy: 0.5011 157/1563 [==>...........................] - ETA: 1s - loss: 1.3985 - accuracy: 0.5040 196/1563 [==>...........................] - ETA: 1s - loss: 1.3991 - accuracy: 0.5037 234/1563 [===>..........................] - ETA: 1s - loss: 1.3994 - accuracy: 0.5041 272/1563 [====>.........................] - ETA: 1s - loss: 1.3991 - accuracy: 0.5043 310/1563 [====>.........................] - ETA: 1s - loss: 1.4032 - accuracy: 0.5032 348/1563 [=====>........................] - ETA: 1s - loss: 1.4016 - accuracy: 0.5052 386/1563 [======>.......................] - ETA: 1s - loss: 1.3996 - accuracy: 0.5065 424/1563 [=======>......................] - ETA: 1s - loss: 1.3990 - accuracy: 0.5060 462/1563 [=======>......................] - ETA: 1s - loss: 1.4014 - accuracy: 0.5049 500/1563 [========>.....................] - ETA: 1s - loss: 1.4015 - accuracy: 0.5060 538/1563 [=========>....................] - ETA: 1s - loss: 1.3975 - accuracy: 0.5067 576/1563 [==========>...................] - ETA: 1s - loss: 1.3954 - accuracy: 0.5079 614/1563 [==========>...................] - ETA: 1s - loss: 1.3964 - accuracy: 0.5079 652/1563 [===========>..................] - ETA: 1s - loss: 1.3974 - accuracy: 0.5085 690/1563 [============>.................] - ETA: 1s - loss: 1.3977 - accuracy: 0.5078 728/1563 [============>.................] - ETA: 1s - loss: 1.3961 - accuracy: 0.5091 766/1563 [=============>................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5091 804/1563 [==============>...............] - ETA: 1s - loss: 1.3966 - accuracy: 0.5081 842/1563 [===============>..............] - ETA: 0s - loss: 1.3959 - accuracy: 0.5087 880/1563 [===============>..............] - ETA: 0s - loss: 1.3951 - accuracy: 0.5093 918/1563 [================>.............] - ETA: 0s - loss: 1.3946 - accuracy: 0.5096 956/1563 [=================>............] - ETA: 0s - loss: 1.3932 - accuracy: 0.5101 994/1563 [==================>...........] - ETA: 0s - loss: 1.3932 - accuracy: 0.5097 1032/1563 [==================>...........] - ETA: 0s - loss: 1.3937 - accuracy: 0.5100 1070/1563 [===================>..........] - ETA: 0s - loss: 1.3942 - accuracy: 0.5101 1108/1563 [====================>.........] - ETA: 0s - loss: 1.3951 - accuracy: 0.5100 1146/1563 [====================>.........] - ETA: 0s - loss: 1.3957 - accuracy: 0.5095 1184/1563 [=====================>........] - ETA: 0s - loss: 1.3949 - accuracy: 0.5100 1222/1563 [======================>.......] - ETA: 0s - loss: 1.3938 - accuracy: 0.5099 1259/1563 [=======================>......] - ETA: 0s - loss: 1.3929 - accuracy: 0.5100 1297/1563 [=======================>......] - ETA: 0s - loss: 1.3917 - accuracy: 0.5101 1335/1563 [========================>.....] - ETA: 0s - loss: 1.3922 - accuracy: 0.5102 1373/1563 [=========================>....] - ETA: 0s - loss: 1.3920 - accuracy: 0.5105 1411/1563 [==========================>...] - ETA: 0s - loss: 1.3914 - accuracy: 0.5111 1449/1563 [==========================>...] - ETA: 0s - loss: 1.3911 - accuracy: 0.5111 1487/1563 [===========================>..] - ETA: 0s - loss: 1.3913 - accuracy: 0.5108 1526/1563 [============================>.] - ETA: 0s - loss: 1.3910 - accuracy: 0.5110 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3896 - accuracy: 0.5116 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.5042 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.3501 - accuracy: 0.5256 77/1563 [>.............................] - ETA: 1s - loss: 1.3670 - accuracy: 0.5170 116/1563 [=>............................] - ETA: 1s - loss: 1.3605 - accuracy: 0.5167 155/1563 [=>............................] - ETA: 1s - loss: 1.3566 - accuracy: 0.5214 193/1563 [==>...........................] - ETA: 1s - loss: 1.3551 - accuracy: 0.5202 231/1563 [===>..........................] - ETA: 1s - loss: 1.3540 - accuracy: 0.5222 269/1563 [====>.........................] - ETA: 1s - loss: 1.3633 - accuracy: 0.5188 307/1563 [====>.........................] - ETA: 1s - loss: 1.3682 - accuracy: 0.5154 346/1563 [=====>........................] - ETA: 1s - loss: 1.3679 - accuracy: 0.5154 385/1563 [======>.......................] - ETA: 1s - loss: 1.3635 - accuracy: 0.5188 424/1563 [=======>......................] - ETA: 1s - loss: 1.3635 - accuracy: 0.5188 462/1563 [=======>......................] - ETA: 1s - loss: 1.3620 - accuracy: 0.5196 499/1563 [========>.....................] - ETA: 1s - loss: 1.3649 - accuracy: 0.5192 537/1563 [=========>....................] - ETA: 1s - loss: 1.3622 - accuracy: 0.5203 575/1563 [==========>...................] - ETA: 1s - loss: 1.3623 - accuracy: 0.5200 613/1563 [==========>...................] - ETA: 1s - loss: 1.3607 - accuracy: 0.5207 650/1563 [===========>..................] - ETA: 1s - loss: 1.3610 - accuracy: 0.5210 688/1563 [============>.................] - ETA: 1s - loss: 1.3594 - accuracy: 0.5218 727/1563 [============>.................] - ETA: 1s - loss: 1.3577 - accuracy: 0.5230 766/1563 [=============>................] - ETA: 1s - loss: 1.3561 - accuracy: 0.5240 805/1563 [==============>...............] - ETA: 1s - loss: 1.3555 - accuracy: 0.5242 844/1563 [===============>..............] - ETA: 0s - loss: 1.3549 - accuracy: 0.5239 883/1563 [===============>..............] - ETA: 0s - loss: 1.3541 - accuracy: 0.5230 922/1563 [================>.............] - ETA: 0s - loss: 1.3553 - accuracy: 0.5220 959/1563 [=================>............] - ETA: 0s - loss: 1.3559 - accuracy: 0.5215 997/1563 [==================>...........] - ETA: 0s - loss: 1.3532 - accuracy: 0.5226 1035/1563 [==================>...........] - ETA: 0s - loss: 1.3534 - accuracy: 0.5231 1073/1563 [===================>..........] - ETA: 0s - loss: 1.3541 - accuracy: 0.5229 1111/1563 [====================>.........] - ETA: 0s - loss: 1.3552 - accuracy: 0.5225 1149/1563 [=====================>........] - ETA: 0s - loss: 1.3546 - accuracy: 0.5226 1187/1563 [=====================>........] - ETA: 0s - loss: 1.3542 - accuracy: 0.5226 1225/1563 [======================>.......] - ETA: 0s - loss: 1.3530 - accuracy: 0.5229 1263/1563 [=======================>......] - ETA: 0s - loss: 1.3534 - accuracy: 0.5226 1301/1563 [=======================>......] - ETA: 0s - loss: 1.3533 - accuracy: 0.5227 1339/1563 [========================>.....] - ETA: 0s - loss: 1.3532 - accuracy: 0.5230 1377/1563 [=========================>....] - ETA: 0s - loss: 1.3535 - accuracy: 0.5233 1415/1563 [==========================>...] - ETA: 0s - loss: 1.3539 - accuracy: 0.5234 1453/1563 [==========================>...] - ETA: 0s - loss: 1.3538 - accuracy: 0.5235 1491/1563 [===========================>..] - ETA: 0s - loss: 1.3537 - accuracy: 0.5236 1529/1563 [============================>.] - ETA: 0s - loss: 1.3525 - accuracy: 0.5238 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3524 - accuracy: 0.5241 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.1560 - accuracy: 0.6875 39/1563 [..............................] - ETA: 2s - loss: 1.2853 - accuracy: 0.5569 77/1563 [>.............................] - ETA: 1s - loss: 1.3022 - accuracy: 0.5483 115/1563 [=>............................] - ETA: 1s - loss: 1.2954 - accuracy: 0.5481 153/1563 [=>............................] - ETA: 1s - loss: 1.3080 - accuracy: 0.5417 191/1563 [==>...........................] - ETA: 1s - loss: 1.3152 - accuracy: 0.5398 229/1563 [===>..........................] - ETA: 1s - loss: 1.3145 - accuracy: 0.5370 267/1563 [====>.........................] - ETA: 1s - loss: 1.3144 - accuracy: 0.5406 305/1563 [====>.........................] - ETA: 1s - loss: 1.3080 - accuracy: 0.5417 343/1563 [=====>........................] - ETA: 1s - loss: 1.3119 - accuracy: 0.5409 381/1563 [======>.......................] - ETA: 1s - loss: 1.3135 - accuracy: 0.5404 419/1563 [=======>......................] - ETA: 1s - loss: 1.3105 - accuracy: 0.5392 457/1563 [=======>......................] - ETA: 1s - loss: 1.3110 - accuracy: 0.5388 495/1563 [========>.....................] - ETA: 1s - loss: 1.3164 - accuracy: 0.5374 533/1563 [=========>....................] - ETA: 1s - loss: 1.3155 - accuracy: 0.5371 571/1563 [=========>....................] - ETA: 1s - loss: 1.3191 - accuracy: 0.5361 609/1563 [==========>...................] - ETA: 1s - loss: 1.3180 - accuracy: 0.5375 647/1563 [===========>..................] - ETA: 1s - loss: 1.3168 - accuracy: 0.5389 685/1563 [============>.................] - ETA: 1s - loss: 1.3145 - accuracy: 0.5392 723/1563 [============>.................] - ETA: 1s - loss: 1.3136 - accuracy: 0.5395 761/1563 [=============>................] - ETA: 1s - loss: 1.3166 - accuracy: 0.5381 799/1563 [==============>...............] - ETA: 1s - loss: 1.3160 - accuracy: 0.5379 837/1563 [===============>..............] - ETA: 0s - loss: 1.3158 - accuracy: 0.5376 874/1563 [===============>..............] - ETA: 0s - loss: 1.3172 - accuracy: 0.5373 912/1563 [================>.............] - ETA: 0s - loss: 1.3187 - accuracy: 0.5366 950/1563 [=================>............] - ETA: 0s - loss: 1.3197 - accuracy: 0.5357 988/1563 [=================>............] - ETA: 0s - loss: 1.3194 - accuracy: 0.5357 1026/1563 [==================>...........] - ETA: 0s - loss: 1.3189 - accuracy: 0.5351 1064/1563 [===================>..........] - ETA: 0s - loss: 1.3186 - accuracy: 0.5352 1102/1563 [====================>.........] - ETA: 0s - loss: 1.3182 - accuracy: 0.5350 1139/1563 [====================>.........] - ETA: 0s - loss: 1.3175 - accuracy: 0.5350 1177/1563 [=====================>........] - ETA: 0s - loss: 1.3175 - accuracy: 0.5350 1215/1563 [======================>.......] - ETA: 0s - loss: 1.3175 - accuracy: 0.5349 1253/1563 [=======================>......] - ETA: 0s - loss: 1.3171 - accuracy: 0.5352 1290/1563 [=======================>......] - ETA: 0s - loss: 1.3178 - accuracy: 0.5350 1328/1563 [========================>.....] - ETA: 0s - loss: 1.3180 - accuracy: 0.5349 1366/1563 [=========================>....] - ETA: 0s - loss: 1.3159 - accuracy: 0.5356 1404/1563 [=========================>....] - ETA: 0s - loss: 1.3158 - accuracy: 0.5356 1442/1563 [==========================>...] - ETA: 0s - loss: 1.3155 - accuracy: 0.5353 1480/1563 [===========================>..] - ETA: 0s - loss: 1.3160 - accuracy: 0.5351 1518/1563 [============================>.] - ETA: 0s - loss: 1.3160 - accuracy: 0.5349 1556/1563 [============================>.] - ETA: 0s - loss: 1.3152 - accuracy: 0.5353 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3152 - accuracy: 0.5353 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.1881 - accuracy: 0.5938 39/1563 [..............................] - ETA: 2s - loss: 1.2589 - accuracy: 0.5609 77/1563 [>.............................] - ETA: 1s - loss: 1.2636 - accuracy: 0.5584 115/1563 [=>............................] - ETA: 1s - loss: 1.2748 - accuracy: 0.5497 153/1563 [=>............................] - ETA: 1s - loss: 1.2677 - accuracy: 0.5476 191/1563 [==>...........................] - ETA: 1s - loss: 1.2817 - accuracy: 0.5468 229/1563 [===>..........................] - ETA: 1s - loss: 1.2743 - accuracy: 0.5490 267/1563 [====>.........................] - ETA: 1s - loss: 1.2710 - accuracy: 0.5506 305/1563 [====>.........................] - ETA: 1s - loss: 1.2680 - accuracy: 0.5504 343/1563 [=====>........................] - ETA: 1s - loss: 1.2729 - accuracy: 0.5498 381/1563 [======>.......................] - ETA: 1s - loss: 1.2756 - accuracy: 0.5493 419/1563 [=======>......................] - ETA: 1s - loss: 1.2754 - accuracy: 0.5500 457/1563 [=======>......................] - ETA: 1s - loss: 1.2742 - accuracy: 0.5512 495/1563 [========>.....................] - ETA: 1s - loss: 1.2761 - accuracy: 0.5509 533/1563 [=========>....................] - ETA: 1s - loss: 1.2743 - accuracy: 0.5530 571/1563 [=========>....................] - ETA: 1s - loss: 1.2739 - accuracy: 0.5533 610/1563 [==========>...................] - ETA: 1s - loss: 1.2758 - accuracy: 0.5519 648/1563 [===========>..................] - ETA: 1s - loss: 1.2777 - accuracy: 0.5518 686/1563 [============>.................] - ETA: 1s - loss: 1.2776 - accuracy: 0.5518 724/1563 [============>.................] - ETA: 1s - loss: 1.2784 - accuracy: 0.5507 762/1563 [=============>................] - ETA: 1s - loss: 1.2801 - accuracy: 0.5493 800/1563 [==============>...............] - ETA: 1s - loss: 1.2788 - accuracy: 0.5496 838/1563 [===============>..............] - ETA: 0s - loss: 1.2771 - accuracy: 0.5508 876/1563 [===============>..............] - ETA: 0s - loss: 1.2774 - accuracy: 0.5514 914/1563 [================>.............] - ETA: 0s - loss: 1.2781 - accuracy: 0.5515 952/1563 [=================>............] - ETA: 0s - loss: 1.2781 - accuracy: 0.5514 990/1563 [==================>...........] - ETA: 0s - loss: 1.2776 - accuracy: 0.5520 1028/1563 [==================>...........] - ETA: 0s - loss: 1.2784 - accuracy: 0.5517 1066/1563 [===================>..........] - ETA: 0s - loss: 1.2798 - accuracy: 0.5505 1104/1563 [====================>.........] - ETA: 0s - loss: 1.2814 - accuracy: 0.5503 1142/1563 [====================>.........] - ETA: 0s - loss: 1.2812 - accuracy: 0.5508 1180/1563 [=====================>........] - ETA: 0s - loss: 1.2821 - accuracy: 0.5505 1218/1563 [======================>.......] - ETA: 0s - loss: 1.2804 - accuracy: 0.5512 1256/1563 [=======================>......] - ETA: 0s - loss: 1.2793 - accuracy: 0.5510 1294/1563 [=======================>......] - ETA: 0s - loss: 1.2791 - accuracy: 0.5513 1332/1563 [========================>.....] - ETA: 0s - loss: 1.2782 - accuracy: 0.5520 1370/1563 [=========================>....] - ETA: 0s - loss: 1.2810 - accuracy: 0.5512 1408/1563 [==========================>...] - ETA: 0s - loss: 1.2824 - accuracy: 0.5508 1445/1563 [==========================>...] - ETA: 0s - loss: 1.2833 - accuracy: 0.5507 1483/1563 [===========================>..] - ETA: 0s - loss: 1.2842 - accuracy: 0.5504 1521/1563 [============================>.] - ETA: 0s - loss: 1.2843 - accuracy: 0.5503 1559/1563 [============================>.] - ETA: 0s - loss: 1.2839 - accuracy: 0.5502 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2836 - accuracy: 0.5504 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.1894 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.1852 - accuracy: 0.5817 77/1563 [>.............................] - ETA: 1s - loss: 1.2167 - accuracy: 0.5787 115/1563 [=>............................] - ETA: 1s - loss: 1.2289 - accuracy: 0.5717 153/1563 [=>............................] - ETA: 1s - loss: 1.2337 - accuracy: 0.5699 191/1563 [==>...........................] - ETA: 1s - loss: 1.2442 - accuracy: 0.5653 229/1563 [===>..........................] - ETA: 1s - loss: 1.2419 - accuracy: 0.5681 267/1563 [====>.........................] - ETA: 1s - loss: 1.2499 - accuracy: 0.5648 305/1563 [====>.........................] - ETA: 1s - loss: 1.2481 - accuracy: 0.5654 343/1563 [=====>........................] - ETA: 1s - loss: 1.2481 - accuracy: 0.5639 381/1563 [======>.......................] - ETA: 1s - loss: 1.2497 - accuracy: 0.5622 419/1563 [=======>......................] - ETA: 1s - loss: 1.2461 - accuracy: 0.5626 457/1563 [=======>......................] - ETA: 1s - loss: 1.2430 - accuracy: 0.5643 494/1563 [========>.....................] - ETA: 1s - loss: 1.2459 - accuracy: 0.5623 531/1563 [=========>....................] - ETA: 1s - loss: 1.2492 - accuracy: 0.5604 569/1563 [=========>....................] - ETA: 1s - loss: 1.2509 - accuracy: 0.5603 607/1563 [==========>...................] - ETA: 1s - loss: 1.2501 - accuracy: 0.5613 645/1563 [===========>..................] - ETA: 1s - loss: 1.2491 - accuracy: 0.5615 683/1563 [============>.................] - ETA: 1s - loss: 1.2475 - accuracy: 0.5625 721/1563 [============>.................] - ETA: 1s - loss: 1.2481 - accuracy: 0.5622 759/1563 [=============>................] - ETA: 1s - loss: 1.2506 - accuracy: 0.5607 797/1563 [==============>...............] - ETA: 1s - loss: 1.2497 - accuracy: 0.5610 835/1563 [===============>..............] - ETA: 0s - loss: 1.2504 - accuracy: 0.5605 873/1563 [===============>..............] - ETA: 0s - loss: 1.2514 - accuracy: 0.5597 911/1563 [================>.............] - ETA: 0s - loss: 1.2525 - accuracy: 0.5597 949/1563 [=================>............] - ETA: 0s - loss: 1.2519 - accuracy: 0.5602 987/1563 [=================>............] - ETA: 0s - loss: 1.2523 - accuracy: 0.5596 1025/1563 [==================>...........] - ETA: 0s - loss: 1.2515 - accuracy: 0.5599 1063/1563 [===================>..........] - ETA: 0s - loss: 1.2517 - accuracy: 0.5603 1101/1563 [====================>.........] - ETA: 0s - loss: 1.2529 - accuracy: 0.5601 1139/1563 [====================>.........] - ETA: 0s - loss: 1.2537 - accuracy: 0.5598 1177/1563 [=====================>........] - ETA: 0s - loss: 1.2545 - accuracy: 0.5591 1215/1563 [======================>.......] - ETA: 0s - loss: 1.2537 - accuracy: 0.5596 1253/1563 [=======================>......] - ETA: 0s - loss: 1.2548 - accuracy: 0.5594 1291/1563 [=======================>......] - ETA: 0s - loss: 1.2544 - accuracy: 0.5597 1328/1563 [========================>.....] - ETA: 0s - loss: 1.2535 - accuracy: 0.5600 1366/1563 [=========================>....] - ETA: 0s - loss: 1.2530 - accuracy: 0.5601 1404/1563 [=========================>....] - ETA: 0s - loss: 1.2543 - accuracy: 0.5602 1442/1563 [==========================>...] - ETA: 0s - loss: 1.2551 - accuracy: 0.5601 1480/1563 [===========================>..] - ETA: 0s - loss: 1.2544 - accuracy: 0.5603 1519/1563 [============================>.] - ETA: 0s - loss: 1.2554 - accuracy: 0.5598 1557/1563 [============================>.] - ETA: 0s - loss: 1.2544 - accuracy: 0.5602 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2537 - accuracy: 0.5603 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 35.74362897872925 hydro05 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins16550735077287292567.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS