Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 96747 queued and waiting for resources srun: job 96747 has been allocated resources 2024-11-05 07:43:06.779463: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-05 07:43:28.496641: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-05 07:43:58.713108: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-05 07:43:58.713663: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-05 07:43:58.713678: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-05 07:44:43.394703: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-05 07:44:45.846660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-05 07:44:53.689130: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:44:44 - loss: 2.3815 - accuracy: 0.0312 38/1563 [..............................] - ETA: 2s - loss: 2.2483 - accuracy: 0.1653  77/1563 [>.............................] - ETA: 1s - loss: 2.1781 - accuracy: 0.1936 116/1563 [=>............................] - ETA: 1s - loss: 2.1220 - accuracy: 0.2214 155/1563 [=>............................] - ETA: 1s - loss: 2.0914 - accuracy: 0.2361 194/1563 [==>...........................] - ETA: 1s - loss: 2.0632 - accuracy: 0.2492 233/1563 [===>..........................] - ETA: 1s - loss: 2.0433 - accuracy: 0.2546 271/1563 [====>.........................] - ETA: 1s - loss: 2.0210 - accuracy: 0.2650 310/1563 [====>.........................] - ETA: 1s - loss: 2.0056 - accuracy: 0.2711 349/1563 [=====>........................] - ETA: 1s - loss: 1.9936 - accuracy: 0.2783 388/1563 [======>.......................] - ETA: 1s - loss: 1.9784 - accuracy: 0.2841 427/1563 [=======>......................] - ETA: 1s - loss: 1.9678 - accuracy: 0.2900 466/1563 [=======>......................] - ETA: 1s - loss: 1.9538 - accuracy: 0.2959 506/1563 [========>.....................] - ETA: 1s - loss: 1.9429 - accuracy: 0.2998 545/1563 [=========>....................] - ETA: 1s - loss: 1.9339 - accuracy: 0.3035 584/1563 [==========>...................] - ETA: 1s - loss: 1.9248 - accuracy: 0.3089 624/1563 [==========>...................] - ETA: 1s - loss: 1.9145 - accuracy: 0.3126 664/1563 [===========>..................] - ETA: 1s - loss: 1.9079 - accuracy: 0.3149 703/1563 [============>.................] - ETA: 1s - loss: 1.8989 - accuracy: 0.3185 743/1563 [=============>................] - ETA: 1s - loss: 1.8937 - accuracy: 0.3208 782/1563 [==============>...............] - ETA: 1s - loss: 1.8890 - accuracy: 0.3223 821/1563 [==============>...............] - ETA: 0s - loss: 1.8851 - accuracy: 0.3250 861/1563 [===============>..............] - ETA: 0s - loss: 1.8794 - accuracy: 0.3272 900/1563 [================>.............] - ETA: 0s - loss: 1.8736 - accuracy: 0.3288 939/1563 [=================>............] - ETA: 0s - loss: 1.8690 - accuracy: 0.3307 978/1563 [=================>............] - ETA: 0s - loss: 1.8625 - accuracy: 0.3338 1017/1563 [==================>...........] - ETA: 0s - loss: 1.8574 - accuracy: 0.3366 1056/1563 [===================>..........] - ETA: 0s - loss: 1.8533 - accuracy: 0.3376 1096/1563 [====================>.........] - ETA: 0s - loss: 1.8485 - accuracy: 0.3393 1136/1563 [====================>.........] - ETA: 0s - loss: 1.8443 - accuracy: 0.3412 1175/1563 [=====================>........] - ETA: 0s - loss: 1.8399 - accuracy: 0.3433 1215/1563 [======================>.......] - ETA: 0s - loss: 1.8381 - accuracy: 0.3442 1255/1563 [=======================>......] - ETA: 0s - loss: 1.8342 - accuracy: 0.3458 1295/1563 [=======================>......] - ETA: 0s - loss: 1.8308 - accuracy: 0.3472 1335/1563 [========================>.....] - ETA: 0s - loss: 1.8278 - accuracy: 0.3485 1375/1563 [=========================>....] - ETA: 0s - loss: 1.8243 - accuracy: 0.3499 1415/1563 [==========================>...] - ETA: 0s - loss: 1.8220 - accuracy: 0.3511 1454/1563 [==========================>...] - ETA: 0s - loss: 1.8182 - accuracy: 0.3529 1494/1563 [===========================>..] - ETA: 0s - loss: 1.8150 - accuracy: 0.3539 1533/1563 [============================>.] - ETA: 0s - loss: 1.8123 - accuracy: 0.3546 1563/1563 [==============================] - 6s 1ms/step - loss: 1.8106 - accuracy: 0.3550 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.4349 - accuracy: 0.5000 41/1563 [..............................] - ETA: 1s - loss: 1.6196 - accuracy: 0.4405 81/1563 [>.............................] - ETA: 1s - loss: 1.6565 - accuracy: 0.4232 121/1563 [=>............................] - ETA: 1s - loss: 1.6517 - accuracy: 0.4192 161/1563 [==>...........................] - ETA: 1s - loss: 1.6585 - accuracy: 0.4140 201/1563 [==>...........................] - ETA: 1s - loss: 1.6583 - accuracy: 0.4109 241/1563 [===>..........................] - ETA: 1s - loss: 1.6557 - accuracy: 0.4109 280/1563 [====>.........................] - ETA: 1s - loss: 1.6479 - accuracy: 0.4134 320/1563 [=====>........................] - ETA: 1s - loss: 1.6447 - accuracy: 0.4159 360/1563 [=====>........................] - ETA: 1s - loss: 1.6451 - accuracy: 0.4156 400/1563 [======>.......................] - ETA: 1s - loss: 1.6442 - accuracy: 0.4155 440/1563 [=======>......................] - ETA: 1s - loss: 1.6441 - accuracy: 0.4170 480/1563 [========>.....................] - ETA: 1s - loss: 1.6455 - accuracy: 0.4178 520/1563 [========>.....................] - ETA: 1s - loss: 1.6464 - accuracy: 0.4175 560/1563 [=========>....................] - ETA: 1s - loss: 1.6475 - accuracy: 0.4181 600/1563 [==========>...................] - ETA: 1s - loss: 1.6466 - accuracy: 0.4182 640/1563 [===========>..................] - ETA: 1s - loss: 1.6428 - accuracy: 0.4198 680/1563 [============>.................] - ETA: 1s - loss: 1.6434 - accuracy: 0.4197 720/1563 [============>.................] - ETA: 1s - loss: 1.6425 - accuracy: 0.4195 760/1563 [=============>................] - ETA: 1s - loss: 1.6409 - accuracy: 0.4201 800/1563 [==============>...............] - ETA: 0s - loss: 1.6395 - accuracy: 0.4200 840/1563 [===============>..............] - ETA: 0s - loss: 1.6376 - accuracy: 0.4208 880/1563 [===============>..............] - ETA: 0s - loss: 1.6364 - accuracy: 0.4199 920/1563 [================>.............] - ETA: 0s - loss: 1.6370 - accuracy: 0.4208 960/1563 [=================>............] - ETA: 0s - loss: 1.6373 - accuracy: 0.4210 1000/1563 [==================>...........] - ETA: 0s - loss: 1.6361 - accuracy: 0.4217 1040/1563 [==================>...........] - ETA: 0s - loss: 1.6352 - accuracy: 0.4219 1080/1563 [===================>..........] - ETA: 0s - loss: 1.6331 - accuracy: 0.4231 1119/1563 [====================>.........] - ETA: 0s - loss: 1.6322 - accuracy: 0.4238 1158/1563 [=====================>........] - ETA: 0s - loss: 1.6319 - accuracy: 0.4233 1198/1563 [=====================>........] - ETA: 0s - loss: 1.6315 - accuracy: 0.4232 1237/1563 [======================>.......] - ETA: 0s - loss: 1.6306 - accuracy: 0.4235 1277/1563 [=======================>......] - ETA: 0s - loss: 1.6299 - accuracy: 0.4242 1317/1563 [========================>.....] - ETA: 0s - loss: 1.6298 - accuracy: 0.4245 1357/1563 [=========================>....] - ETA: 0s - loss: 1.6283 - accuracy: 0.4249 1397/1563 [=========================>....] - ETA: 0s - loss: 1.6270 - accuracy: 0.4254 1436/1563 [==========================>...] - ETA: 0s - loss: 1.6261 - accuracy: 0.4258 1475/1563 [===========================>..] - ETA: 0s - loss: 1.6247 - accuracy: 0.4259 1515/1563 [============================>.] - ETA: 0s - loss: 1.6239 - accuracy: 0.4263 1554/1563 [============================>.] - ETA: 0s - loss: 1.6224 - accuracy: 0.4267 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6223 - accuracy: 0.4268 Epoch 3/10 1/1563 [..............................] - ETA: 1s - loss: 1.4495 - accuracy: 0.4375 41/1563 [..............................] - ETA: 1s - loss: 1.5587 - accuracy: 0.4421 81/1563 [>.............................] - ETA: 1s - loss: 1.5533 - accuracy: 0.4452 121/1563 [=>............................] - ETA: 1s - loss: 1.5486 - accuracy: 0.4378 161/1563 [==>...........................] - ETA: 1s - loss: 1.5580 - accuracy: 0.4356 201/1563 [==>...........................] - ETA: 1s - loss: 1.5593 - accuracy: 0.4366 241/1563 [===>..........................] - ETA: 1s - loss: 1.5667 - accuracy: 0.4324 280/1563 [====>.........................] - ETA: 1s - loss: 1.5671 - accuracy: 0.4323 318/1563 [=====>........................] - ETA: 1s - loss: 1.5706 - accuracy: 0.4308 358/1563 [=====>........................] - ETA: 1s - loss: 1.5674 - accuracy: 0.4351 398/1563 [======>.......................] - ETA: 1s - loss: 1.5680 - accuracy: 0.4362 438/1563 [=======>......................] - ETA: 1s - loss: 1.5652 - accuracy: 0.4375 477/1563 [========>.....................] - ETA: 1s - loss: 1.5660 - accuracy: 0.4365 517/1563 [========>.....................] - ETA: 1s - loss: 1.5643 - accuracy: 0.4395 557/1563 [=========>....................] - ETA: 1s - loss: 1.5638 - accuracy: 0.4391 597/1563 [==========>...................] - ETA: 1s - loss: 1.5599 - accuracy: 0.4421 637/1563 [===========>..................] - ETA: 1s - loss: 1.5574 - accuracy: 0.4438 676/1563 [===========>..................] - ETA: 1s - loss: 1.5569 - accuracy: 0.4441 715/1563 [============>.................] - ETA: 1s - loss: 1.5550 - accuracy: 0.4461 755/1563 [=============>................] - ETA: 1s - loss: 1.5534 - accuracy: 0.4474 794/1563 [==============>...............] - ETA: 0s - loss: 1.5504 - accuracy: 0.4489 834/1563 [===============>..............] - ETA: 0s - loss: 1.5513 - accuracy: 0.4493 873/1563 [===============>..............] - ETA: 0s - loss: 1.5528 - accuracy: 0.4491 912/1563 [================>.............] - ETA: 0s - loss: 1.5529 - accuracy: 0.4492 951/1563 [=================>............] - ETA: 0s - loss: 1.5504 - accuracy: 0.4495 990/1563 [==================>...........] - ETA: 0s - loss: 1.5511 - accuracy: 0.4501 1029/1563 [==================>...........] - ETA: 0s - loss: 1.5499 - accuracy: 0.4507 1068/1563 [===================>..........] - ETA: 0s - loss: 1.5487 - accuracy: 0.4518 1107/1563 [====================>.........] - ETA: 0s - loss: 1.5479 - accuracy: 0.4527 1146/1563 [====================>.........] - ETA: 0s - loss: 1.5466 - accuracy: 0.4532 1185/1563 [=====================>........] - ETA: 0s - loss: 1.5450 - accuracy: 0.4536 1224/1563 [======================>.......] - ETA: 0s - loss: 1.5444 - accuracy: 0.4542 1263/1563 [=======================>......] - ETA: 0s - loss: 1.5417 - accuracy: 0.4552 1303/1563 [========================>.....] - ETA: 0s - loss: 1.5414 - accuracy: 0.4555 1342/1563 [========================>.....] - ETA: 0s - loss: 1.5419 - accuracy: 0.4555 1381/1563 [=========================>....] - ETA: 0s - loss: 1.5404 - accuracy: 0.4560 1420/1563 [==========================>...] - ETA: 0s - loss: 1.5406 - accuracy: 0.4563 1459/1563 [===========================>..] - ETA: 0s - loss: 1.5413 - accuracy: 0.4563 1497/1563 [===========================>..] - ETA: 0s - loss: 1.5406 - accuracy: 0.4564 1537/1563 [============================>.] - ETA: 0s - loss: 1.5407 - accuracy: 0.4562 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5416 - accuracy: 0.4562 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.3514 - accuracy: 0.5625 41/1563 [..............................] - ETA: 1s - loss: 1.5309 - accuracy: 0.4512 81/1563 [>.............................] - ETA: 1s - loss: 1.5130 - accuracy: 0.4633 121/1563 [=>............................] - ETA: 1s - loss: 1.4942 - accuracy: 0.4677 162/1563 [==>...........................] - ETA: 1s - loss: 1.4858 - accuracy: 0.4709 202/1563 [==>...........................] - ETA: 1s - loss: 1.4822 - accuracy: 0.4773 242/1563 [===>..........................] - ETA: 1s - loss: 1.4850 - accuracy: 0.4771 282/1563 [====>.........................] - ETA: 1s - loss: 1.4795 - accuracy: 0.4779 323/1563 [=====>........................] - ETA: 1s - loss: 1.4791 - accuracy: 0.4787 363/1563 [=====>........................] - ETA: 1s - loss: 1.4861 - accuracy: 0.4767 403/1563 [======>.......................] - ETA: 1s - loss: 1.4883 - accuracy: 0.4753 443/1563 [=======>......................] - ETA: 1s - loss: 1.4899 - accuracy: 0.4735 483/1563 [========>.....................] - ETA: 1s - loss: 1.4897 - accuracy: 0.4739 523/1563 [=========>....................] - ETA: 1s - loss: 1.4888 - accuracy: 0.4738 563/1563 [=========>....................] - ETA: 1s - loss: 1.4888 - accuracy: 0.4741 603/1563 [==========>...................] - ETA: 1s - loss: 1.4873 - accuracy: 0.4748 643/1563 [===========>..................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4740 684/1563 [============>.................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4736 724/1563 [============>.................] - ETA: 1s - loss: 1.4896 - accuracy: 0.4739 764/1563 [=============>................] - ETA: 1s - loss: 1.4900 - accuracy: 0.4741 804/1563 [==============>...............] - ETA: 0s - loss: 1.4910 - accuracy: 0.4738 844/1563 [===============>..............] - ETA: 0s - loss: 1.4936 - accuracy: 0.4723 885/1563 [===============>..............] - ETA: 0s - loss: 1.4906 - accuracy: 0.4737 926/1563 [================>.............] - ETA: 0s - loss: 1.4909 - accuracy: 0.4735 967/1563 [=================>............] - ETA: 0s - loss: 1.4893 - accuracy: 0.4740 1008/1563 [==================>...........] - ETA: 0s - loss: 1.4873 - accuracy: 0.4749 1049/1563 [===================>..........] - ETA: 0s - loss: 1.4850 - accuracy: 0.4758 1090/1563 [===================>..........] - ETA: 0s - loss: 1.4859 - accuracy: 0.4759 1130/1563 [====================>.........] - ETA: 0s - loss: 1.4855 - accuracy: 0.4762 1171/1563 [=====================>........] - ETA: 0s - loss: 1.4849 - accuracy: 0.4763 1211/1563 [======================>.......] - ETA: 0s - loss: 1.4842 - accuracy: 0.4771 1252/1563 [=======================>......] - ETA: 0s - loss: 1.4832 - accuracy: 0.4774 1292/1563 [=======================>......] - ETA: 0s - loss: 1.4820 - accuracy: 0.4779 1332/1563 [========================>.....] - ETA: 0s - loss: 1.4838 - accuracy: 0.4764 1372/1563 [=========================>....] - ETA: 0s - loss: 1.4838 - accuracy: 0.4762 1412/1563 [==========================>...] - ETA: 0s - loss: 1.4824 - accuracy: 0.4769 1452/1563 [==========================>...] - ETA: 0s - loss: 1.4826 - accuracy: 0.4763 1492/1563 [===========================>..] - ETA: 0s - loss: 1.4832 - accuracy: 0.4763 1532/1563 [============================>.] - ETA: 0s - loss: 1.4825 - accuracy: 0.4763 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4816 - accuracy: 0.4764 Epoch 5/10 1/1563 [..............................] - ETA: 1s - loss: 1.6585 - accuracy: 0.3125 41/1563 [..............................] - ETA: 1s - loss: 1.4441 - accuracy: 0.5000 81/1563 [>.............................] - ETA: 1s - loss: 1.4206 - accuracy: 0.5039 121/1563 [=>............................] - ETA: 1s - loss: 1.4242 - accuracy: 0.4972 162/1563 [==>...........................] - ETA: 1s - loss: 1.4310 - accuracy: 0.4931 202/1563 [==>...........................] - ETA: 1s - loss: 1.4387 - accuracy: 0.4910 242/1563 [===>..........................] - ETA: 1s - loss: 1.4445 - accuracy: 0.4919 282/1563 [====>.........................] - ETA: 1s - loss: 1.4419 - accuracy: 0.4955 322/1563 [=====>........................] - ETA: 1s - loss: 1.4413 - accuracy: 0.4928 361/1563 [=====>........................] - ETA: 1s - loss: 1.4432 - accuracy: 0.4932 400/1563 [======>.......................] - ETA: 1s - loss: 1.4426 - accuracy: 0.4937 439/1563 [=======>......................] - ETA: 1s - loss: 1.4425 - accuracy: 0.4937 478/1563 [========>.....................] - ETA: 1s - loss: 1.4432 - accuracy: 0.4927 518/1563 [========>.....................] - ETA: 1s - loss: 1.4434 - accuracy: 0.4928 557/1563 [=========>....................] - ETA: 1s - loss: 1.4437 - accuracy: 0.4924 596/1563 [==========>...................] - ETA: 1s - loss: 1.4455 - accuracy: 0.4924 635/1563 [===========>..................] - ETA: 1s - loss: 1.4472 - accuracy: 0.4932 675/1563 [===========>..................] - ETA: 1s - loss: 1.4474 - accuracy: 0.4924 715/1563 [============>.................] - ETA: 1s - loss: 1.4470 - accuracy: 0.4920 755/1563 [=============>................] - ETA: 1s - loss: 1.4466 - accuracy: 0.4920 794/1563 [==============>...............] - ETA: 0s - loss: 1.4452 - accuracy: 0.4927 833/1563 [==============>...............] - ETA: 0s - loss: 1.4439 - accuracy: 0.4928 872/1563 [===============>..............] - ETA: 0s - loss: 1.4452 - accuracy: 0.4916 911/1563 [================>.............] - ETA: 0s - loss: 1.4439 - accuracy: 0.4926 951/1563 [=================>............] - ETA: 0s - loss: 1.4431 - accuracy: 0.4926 990/1563 [==================>...........] - ETA: 0s - loss: 1.4424 - accuracy: 0.4934 1029/1563 [==================>...........] - ETA: 0s - loss: 1.4424 - accuracy: 0.4937 1068/1563 [===================>..........] - ETA: 0s - loss: 1.4413 - accuracy: 0.4933 1107/1563 [====================>.........] - ETA: 0s - loss: 1.4419 - accuracy: 0.4931 1146/1563 [====================>.........] - ETA: 0s - loss: 1.4405 - accuracy: 0.4936 1186/1563 [=====================>........] - ETA: 0s - loss: 1.4405 - accuracy: 0.4936 1226/1563 [======================>.......] - ETA: 0s - loss: 1.4375 - accuracy: 0.4938 1265/1563 [=======================>......] - ETA: 0s - loss: 1.4366 - accuracy: 0.4944 1304/1563 [========================>.....] - ETA: 0s - loss: 1.4374 - accuracy: 0.4943 1343/1563 [========================>.....] - ETA: 0s - loss: 1.4372 - accuracy: 0.4945 1383/1563 [=========================>....] - ETA: 0s - loss: 1.4371 - accuracy: 0.4946 1422/1563 [==========================>...] - ETA: 0s - loss: 1.4362 - accuracy: 0.4947 1461/1563 [===========================>..] - ETA: 0s - loss: 1.4354 - accuracy: 0.4947 1500/1563 [===========================>..] - ETA: 0s - loss: 1.4355 - accuracy: 0.4945 1540/1563 [============================>.] - ETA: 0s - loss: 1.4342 - accuracy: 0.4950 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4336 - accuracy: 0.4954 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.3733 - accuracy: 0.5312 40/1563 [..............................] - ETA: 1s - loss: 1.4504 - accuracy: 0.4969 79/1563 [>.............................] - ETA: 1s - loss: 1.4157 - accuracy: 0.5095 119/1563 [=>............................] - ETA: 1s - loss: 1.4209 - accuracy: 0.5055 158/1563 [==>...........................] - ETA: 1s - loss: 1.4255 - accuracy: 0.5061 197/1563 [==>...........................] - ETA: 1s - loss: 1.4168 - accuracy: 0.5119 236/1563 [===>..........................] - ETA: 1s - loss: 1.4142 - accuracy: 0.5093 275/1563 [====>.........................] - ETA: 1s - loss: 1.4046 - accuracy: 0.5120 314/1563 [=====>........................] - ETA: 1s - loss: 1.3990 - accuracy: 0.5142 354/1563 [=====>........................] - ETA: 1s - loss: 1.4007 - accuracy: 0.5132 393/1563 [======>.......................] - ETA: 1s - loss: 1.4001 - accuracy: 0.5115 432/1563 [=======>......................] - ETA: 1s - loss: 1.3984 - accuracy: 0.5116 471/1563 [========>.....................] - ETA: 1s - loss: 1.3959 - accuracy: 0.5117 510/1563 [========>.....................] - ETA: 1s - loss: 1.3937 - accuracy: 0.5121 549/1563 [=========>....................] - ETA: 1s - loss: 1.3913 - accuracy: 0.5134 588/1563 [==========>...................] - ETA: 1s - loss: 1.3929 - accuracy: 0.5129 627/1563 [===========>..................] - ETA: 1s - loss: 1.3955 - accuracy: 0.5118 668/1563 [===========>..................] - ETA: 1s - loss: 1.3960 - accuracy: 0.5111 707/1563 [============>.................] - ETA: 1s - loss: 1.3959 - accuracy: 0.5109 747/1563 [=============>................] - ETA: 1s - loss: 1.3970 - accuracy: 0.5105 788/1563 [==============>...............] - ETA: 0s - loss: 1.3958 - accuracy: 0.5106 829/1563 [==============>...............] - ETA: 0s - loss: 1.3962 - accuracy: 0.5097 870/1563 [===============>..............] - ETA: 0s - loss: 1.3964 - accuracy: 0.5100 910/1563 [================>.............] - ETA: 0s - loss: 1.3953 - accuracy: 0.5111 950/1563 [=================>............] - ETA: 0s - loss: 1.3940 - accuracy: 0.5114 990/1563 [==================>...........] - ETA: 0s - loss: 1.3957 - accuracy: 0.5109 1030/1563 [==================>...........] - ETA: 0s - loss: 1.3929 - accuracy: 0.5112 1070/1563 [===================>..........] - ETA: 0s - loss: 1.3941 - accuracy: 0.5103 1111/1563 [====================>.........] - ETA: 0s - loss: 1.3928 - accuracy: 0.5115 1152/1563 [=====================>........] - ETA: 0s - loss: 1.3937 - accuracy: 0.5111 1193/1563 [=====================>........] - ETA: 0s - loss: 1.3935 - accuracy: 0.5111 1233/1563 [======================>.......] - ETA: 0s - loss: 1.3920 - accuracy: 0.5117 1273/1563 [=======================>......] - ETA: 0s - loss: 1.3918 - accuracy: 0.5119 1313/1563 [========================>.....] - ETA: 0s - loss: 1.3919 - accuracy: 0.5120 1354/1563 [========================>.....] - ETA: 0s - loss: 1.3928 - accuracy: 0.5116 1394/1563 [=========================>....] - ETA: 0s - loss: 1.3934 - accuracy: 0.5108 1434/1563 [==========================>...] - ETA: 0s - loss: 1.3929 - accuracy: 0.5103 1474/1563 [===========================>..] - ETA: 0s - loss: 1.3915 - accuracy: 0.5109 1514/1563 [============================>.] - ETA: 0s - loss: 1.3919 - accuracy: 0.5105 1554/1563 [============================>.] - ETA: 0s - loss: 1.3904 - accuracy: 0.5113 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3901 - accuracy: 0.5114 Epoch 7/10 1/1563 [..............................] - ETA: 1s - loss: 1.0698 - accuracy: 0.5625 41/1563 [..............................] - ETA: 1s - loss: 1.3475 - accuracy: 0.5236 81/1563 [>.............................] - ETA: 1s - loss: 1.3599 - accuracy: 0.5193 121/1563 [=>............................] - ETA: 1s - loss: 1.3611 - accuracy: 0.5145 161/1563 [==>...........................] - ETA: 1s - loss: 1.3555 - accuracy: 0.5159 201/1563 [==>...........................] - ETA: 1s - loss: 1.3579 - accuracy: 0.5173 241/1563 [===>..........................] - ETA: 1s - loss: 1.3595 - accuracy: 0.5205 281/1563 [====>.........................] - ETA: 1s - loss: 1.3550 - accuracy: 0.5202 321/1563 [=====>........................] - ETA: 1s - loss: 1.3581 - accuracy: 0.5193 361/1563 [=====>........................] - ETA: 1s - loss: 1.3565 - accuracy: 0.5195 401/1563 [======>.......................] - ETA: 1s - loss: 1.3593 - accuracy: 0.5207 441/1563 [=======>......................] - ETA: 1s - loss: 1.3606 - accuracy: 0.5203 480/1563 [========>.....................] - ETA: 1s - loss: 1.3601 - accuracy: 0.5184 520/1563 [========>.....................] - ETA: 1s - loss: 1.3602 - accuracy: 0.5188 560/1563 [=========>....................] - ETA: 1s - loss: 1.3598 - accuracy: 0.5186 600/1563 [==========>...................] - ETA: 1s - loss: 1.3567 - accuracy: 0.5206 640/1563 [===========>..................] - ETA: 1s - loss: 1.3570 - accuracy: 0.5209 680/1563 [============>.................] - ETA: 1s - loss: 1.3566 - accuracy: 0.5211 721/1563 [============>.................] - ETA: 1s - loss: 1.3574 - accuracy: 0.5220 761/1563 [=============>................] - ETA: 1s - loss: 1.3550 - accuracy: 0.5233 801/1563 [==============>...............] - ETA: 0s - loss: 1.3541 - accuracy: 0.5234 841/1563 [===============>..............] - ETA: 0s - loss: 1.3552 - accuracy: 0.5223 881/1563 [===============>..............] - ETA: 0s - loss: 1.3552 - accuracy: 0.5231 921/1563 [================>.............] - ETA: 0s - loss: 1.3556 - accuracy: 0.5223 960/1563 [=================>............] - ETA: 0s - loss: 1.3541 - accuracy: 0.5229 1000/1563 [==================>...........] - ETA: 0s - loss: 1.3525 - accuracy: 0.5234 1040/1563 [==================>...........] - ETA: 0s - loss: 1.3515 - accuracy: 0.5235 1080/1563 [===================>..........] - ETA: 0s - loss: 1.3517 - accuracy: 0.5239 1120/1563 [====================>.........] - ETA: 0s - loss: 1.3528 - accuracy: 0.5238 1160/1563 [=====================>........] - ETA: 0s - loss: 1.3533 - accuracy: 0.5241 1200/1563 [======================>.......] - ETA: 0s - loss: 1.3522 - accuracy: 0.5251 1240/1563 [======================>.......] - ETA: 0s - loss: 1.3522 - accuracy: 0.5251 1280/1563 [=======================>......] - ETA: 0s - loss: 1.3529 - accuracy: 0.5248 1320/1563 [========================>.....] - ETA: 0s - loss: 1.3525 - accuracy: 0.5251 1360/1563 [=========================>....] - ETA: 0s - loss: 1.3522 - accuracy: 0.5247 1399/1563 [=========================>....] - ETA: 0s - loss: 1.3515 - accuracy: 0.5246 1439/1563 [==========================>...] - ETA: 0s - loss: 1.3514 - accuracy: 0.5249 1479/1563 [===========================>..] - ETA: 0s - loss: 1.3501 - accuracy: 0.5255 1519/1563 [============================>.] - ETA: 0s - loss: 1.3507 - accuracy: 0.5257 1558/1563 [============================>.] - ETA: 0s - loss: 1.3512 - accuracy: 0.5252 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3513 - accuracy: 0.5252 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.6233 - accuracy: 0.4688 40/1563 [..............................] - ETA: 1s - loss: 1.3084 - accuracy: 0.5414 79/1563 [>.............................] - ETA: 1s - loss: 1.3166 - accuracy: 0.5332 118/1563 [=>............................] - ETA: 1s - loss: 1.3179 - accuracy: 0.5368 157/1563 [==>...........................] - ETA: 1s - loss: 1.3241 - accuracy: 0.5334 196/1563 [==>...........................] - ETA: 1s - loss: 1.3219 - accuracy: 0.5362 235/1563 [===>..........................] - ETA: 1s - loss: 1.3184 - accuracy: 0.5383 274/1563 [====>.........................] - ETA: 1s - loss: 1.3194 - accuracy: 0.5384 313/1563 [=====>........................] - ETA: 1s - loss: 1.3241 - accuracy: 0.5354 352/1563 [=====>........................] - ETA: 1s - loss: 1.3261 - accuracy: 0.5336 391/1563 [======>.......................] - ETA: 1s - loss: 1.3200 - accuracy: 0.5365 430/1563 [=======>......................] - ETA: 1s - loss: 1.3225 - accuracy: 0.5360 469/1563 [========>.....................] - ETA: 1s - loss: 1.3208 - accuracy: 0.5366 507/1563 [========>.....................] - ETA: 1s - loss: 1.3221 - accuracy: 0.5359 546/1563 [=========>....................] - ETA: 1s - loss: 1.3227 - accuracy: 0.5367 585/1563 [==========>...................] - ETA: 1s - loss: 1.3215 - accuracy: 0.5372 625/1563 [==========>...................] - ETA: 1s - loss: 1.3214 - accuracy: 0.5385 665/1563 [===========>..................] - ETA: 1s - loss: 1.3214 - accuracy: 0.5383 705/1563 [============>.................] - ETA: 1s - loss: 1.3223 - accuracy: 0.5372 745/1563 [=============>................] - ETA: 1s - loss: 1.3228 - accuracy: 0.5367 785/1563 [==============>...............] - ETA: 1s - loss: 1.3207 - accuracy: 0.5373 825/1563 [==============>...............] - ETA: 0s - loss: 1.3205 - accuracy: 0.5374 865/1563 [===============>..............] - ETA: 0s - loss: 1.3209 - accuracy: 0.5372 905/1563 [================>.............] - ETA: 0s - loss: 1.3220 - accuracy: 0.5368 945/1563 [=================>............] - ETA: 0s - loss: 1.3234 - accuracy: 0.5365 984/1563 [=================>............] - ETA: 0s - loss: 1.3232 - accuracy: 0.5359 1024/1563 [==================>...........] - ETA: 0s - loss: 1.3239 - accuracy: 0.5354 1064/1563 [===================>..........] - ETA: 0s - loss: 1.3249 - accuracy: 0.5357 1104/1563 [====================>.........] - ETA: 0s - loss: 1.3264 - accuracy: 0.5350 1144/1563 [====================>.........] - ETA: 0s - loss: 1.3258 - accuracy: 0.5355 1184/1563 [=====================>........] - ETA: 0s - loss: 1.3260 - accuracy: 0.5351 1224/1563 [======================>.......] - ETA: 0s - loss: 1.3241 - accuracy: 0.5354 1264/1563 [=======================>......] - ETA: 0s - loss: 1.3219 - accuracy: 0.5358 1304/1563 [========================>.....] - ETA: 0s - loss: 1.3205 - accuracy: 0.5365 1344/1563 [========================>.....] - ETA: 0s - loss: 1.3199 - accuracy: 0.5368 1384/1563 [=========================>....] - ETA: 0s - loss: 1.3174 - accuracy: 0.5373 1424/1563 [==========================>...] - ETA: 0s - loss: 1.3172 - accuracy: 0.5374 1464/1563 [===========================>..] - ETA: 0s - loss: 1.3172 - accuracy: 0.5373 1503/1563 [===========================>..] - ETA: 0s - loss: 1.3177 - accuracy: 0.5373 1542/1563 [============================>.] - ETA: 0s - loss: 1.3174 - accuracy: 0.5373 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3169 - accuracy: 0.5376 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.2670 - accuracy: 0.4062 40/1563 [..............................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5625 78/1563 [>.............................] - ETA: 1s - loss: 1.2670 - accuracy: 0.5613 117/1563 [=>............................] - ETA: 1s - loss: 1.2733 - accuracy: 0.5553 156/1563 [=>............................] - ETA: 1s - loss: 1.2701 - accuracy: 0.5539 194/1563 [==>...........................] - ETA: 1s - loss: 1.2756 - accuracy: 0.5557 233/1563 [===>..........................] - ETA: 1s - loss: 1.2794 - accuracy: 0.5541 272/1563 [====>.........................] - ETA: 1s - loss: 1.2808 - accuracy: 0.5546 311/1563 [====>.........................] - ETA: 1s - loss: 1.2800 - accuracy: 0.5550 349/1563 [=====>........................] - ETA: 1s - loss: 1.2817 - accuracy: 0.5539 388/1563 [======>.......................] - ETA: 1s - loss: 1.2884 - accuracy: 0.5500 427/1563 [=======>......................] - ETA: 1s - loss: 1.2917 - accuracy: 0.5485 466/1563 [=======>......................] - ETA: 1s - loss: 1.2898 - accuracy: 0.5484 505/1563 [========>.....................] - ETA: 1s - loss: 1.2888 - accuracy: 0.5475 544/1563 [=========>....................] - ETA: 1s - loss: 1.2867 - accuracy: 0.5490 583/1563 [==========>...................] - ETA: 1s - loss: 1.2846 - accuracy: 0.5500 622/1563 [==========>...................] - ETA: 1s - loss: 1.2830 - accuracy: 0.5509 661/1563 [===========>..................] - ETA: 1s - loss: 1.2820 - accuracy: 0.5517 700/1563 [============>.................] - ETA: 1s - loss: 1.2822 - accuracy: 0.5510 739/1563 [=============>................] - ETA: 1s - loss: 1.2842 - accuracy: 0.5493 778/1563 [=============>................] - ETA: 1s - loss: 1.2860 - accuracy: 0.5483 817/1563 [==============>...............] - ETA: 0s - loss: 1.2839 - accuracy: 0.5489 856/1563 [===============>..............] - ETA: 0s - loss: 1.2866 - accuracy: 0.5478 895/1563 [================>.............] - ETA: 0s - loss: 1.2851 - accuracy: 0.5473 934/1563 [================>.............] - ETA: 0s - loss: 1.2861 - accuracy: 0.5474 973/1563 [=================>............] - ETA: 0s - loss: 1.2862 - accuracy: 0.5471 1012/1563 [==================>...........] - ETA: 0s - loss: 1.2851 - accuracy: 0.5469 1050/1563 [===================>..........] - ETA: 0s - loss: 1.2838 - accuracy: 0.5478 1089/1563 [===================>..........] - ETA: 0s - loss: 1.2840 - accuracy: 0.5482 1128/1563 [====================>.........] - ETA: 0s - loss: 1.2843 - accuracy: 0.5481 1167/1563 [=====================>........] - ETA: 0s - loss: 1.2846 - accuracy: 0.5479 1206/1563 [======================>.......] - ETA: 0s - loss: 1.2833 - accuracy: 0.5486 1245/1563 [======================>.......] - ETA: 0s - loss: 1.2829 - accuracy: 0.5488 1284/1563 [=======================>......] - ETA: 0s - loss: 1.2819 - accuracy: 0.5491 1323/1563 [========================>.....] - ETA: 0s - loss: 1.2827 - accuracy: 0.5488 1361/1563 [=========================>....] - ETA: 0s - loss: 1.2826 - accuracy: 0.5493 1399/1563 [=========================>....] - ETA: 0s - loss: 1.2837 - accuracy: 0.5490 1436/1563 [==========================>...] - ETA: 0s - loss: 1.2833 - accuracy: 0.5491 1474/1563 [===========================>..] - ETA: 0s - loss: 1.2841 - accuracy: 0.5481 1513/1563 [============================>.] - ETA: 0s - loss: 1.2841 - accuracy: 0.5481 1552/1563 [============================>.] - ETA: 0s - loss: 1.2841 - accuracy: 0.5480 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2836 - accuracy: 0.5481 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.5916 - accuracy: 0.5000 40/1563 [..............................] - ETA: 1s - loss: 1.2121 - accuracy: 0.5758 79/1563 [>.............................] - ETA: 1s - loss: 1.2103 - accuracy: 0.5720 118/1563 [=>............................] - ETA: 1s - loss: 1.2104 - accuracy: 0.5720 157/1563 [==>...........................] - ETA: 1s - loss: 1.2222 - accuracy: 0.5719 196/1563 [==>...........................] - ETA: 1s - loss: 1.2269 - accuracy: 0.5679 235/1563 [===>..........................] - ETA: 1s - loss: 1.2388 - accuracy: 0.5638 273/1563 [====>.........................] - ETA: 1s - loss: 1.2415 - accuracy: 0.5618 312/1563 [====>.........................] - ETA: 1s - loss: 1.2396 - accuracy: 0.5599 351/1563 [=====>........................] - ETA: 1s - loss: 1.2444 - accuracy: 0.5597 390/1563 [======>.......................] - ETA: 1s - loss: 1.2460 - accuracy: 0.5615 429/1563 [=======>......................] - ETA: 1s - loss: 1.2479 - accuracy: 0.5610 468/1563 [=======>......................] - ETA: 1s - loss: 1.2492 - accuracy: 0.5607 507/1563 [========>.....................] - ETA: 1s - loss: 1.2494 - accuracy: 0.5615 546/1563 [=========>....................] - ETA: 1s - loss: 1.2512 - accuracy: 0.5610 585/1563 [==========>...................] - ETA: 1s - loss: 1.2505 - accuracy: 0.5623 624/1563 [==========>...................] - ETA: 1s - loss: 1.2508 - accuracy: 0.5623 663/1563 [===========>..................] - ETA: 1s - loss: 1.2551 - accuracy: 0.5623 702/1563 [============>.................] - ETA: 1s - loss: 1.2536 - accuracy: 0.5625 741/1563 [=============>................] - ETA: 1s - loss: 1.2511 - accuracy: 0.5636 780/1563 [=============>................] - ETA: 1s - loss: 1.2529 - accuracy: 0.5629 819/1563 [==============>...............] - ETA: 0s - loss: 1.2536 - accuracy: 0.5634 858/1563 [===============>..............] - ETA: 0s - loss: 1.2510 - accuracy: 0.5643 897/1563 [================>.............] - ETA: 0s - loss: 1.2513 - accuracy: 0.5640 935/1563 [================>.............] - ETA: 0s - loss: 1.2520 - accuracy: 0.5638 974/1563 [=================>............] - ETA: 0s - loss: 1.2515 - accuracy: 0.5637 1013/1563 [==================>...........] - ETA: 0s - loss: 1.2509 - accuracy: 0.5636 1052/1563 [===================>..........] - ETA: 0s - loss: 1.2513 - accuracy: 0.5637 1091/1563 [===================>..........] - ETA: 0s - loss: 1.2534 - accuracy: 0.5626 1130/1563 [====================>.........] - ETA: 0s - loss: 1.2506 - accuracy: 0.5634 1169/1563 [=====================>........] - ETA: 0s - loss: 1.2517 - accuracy: 0.5621 1208/1563 [======================>.......] - ETA: 0s - loss: 1.2528 - accuracy: 0.5616 1247/1563 [======================>.......] - ETA: 0s - loss: 1.2535 - accuracy: 0.5615 1286/1563 [=======================>......] - ETA: 0s - loss: 1.2512 - accuracy: 0.5626 1325/1563 [========================>.....] - ETA: 0s - loss: 1.2507 - accuracy: 0.5625 1365/1563 [=========================>....] - ETA: 0s - loss: 1.2506 - accuracy: 0.5627 1405/1563 [=========================>....] - ETA: 0s - loss: 1.2499 - accuracy: 0.5636 1445/1563 [==========================>...] - ETA: 0s - loss: 1.2502 - accuracy: 0.5631 1485/1563 [===========================>..] - ETA: 0s - loss: 1.2510 - accuracy: 0.5631 1525/1563 [============================>.] - ETA: 0s - loss: 1.2515 - accuracy: 0.5630 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2507 - accuracy: 0.5634 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 36.206727027893066 hydro04 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins4276250844956316637.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS