Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 96317 queued and waiting for resources srun: job 96317 has been allocated resources 2024-11-01 07:43:07.780981: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-01 07:43:31.216834: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-01 07:44:04.819936: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-01 07:44:04.820513: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-01 07:44:04.820522: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-01 07:44:57.750820: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-01 07:45:00.439793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-01 07:45:07.290975: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:43:15 - loss: 2.4395 - accuracy: 0.1250 38/1563 [..............................] - ETA: 2s - loss: 2.2476 - accuracy: 0.1686  76/1563 [>.............................] - ETA: 2s - loss: 2.1806 - accuracy: 0.1974 114/1563 [=>............................] - ETA: 1s - loss: 2.1261 - accuracy: 0.2201 153/1563 [=>............................] - ETA: 1s - loss: 2.0824 - accuracy: 0.2392 191/1563 [==>...........................] - ETA: 1s - loss: 2.0581 - accuracy: 0.2492 230/1563 [===>..........................] - ETA: 1s - loss: 2.0345 - accuracy: 0.2626 268/1563 [====>.........................] - ETA: 1s - loss: 2.0174 - accuracy: 0.2715 307/1563 [====>.........................] - ETA: 1s - loss: 2.0031 - accuracy: 0.2778 346/1563 [=====>........................] - ETA: 1s - loss: 1.9889 - accuracy: 0.2819 385/1563 [======>.......................] - ETA: 1s - loss: 1.9742 - accuracy: 0.2886 424/1563 [=======>......................] - ETA: 1s - loss: 1.9619 - accuracy: 0.2935 463/1563 [=======>......................] - ETA: 1s - loss: 1.9485 - accuracy: 0.2970 502/1563 [========>.....................] - ETA: 1s - loss: 1.9402 - accuracy: 0.3004 541/1563 [=========>....................] - ETA: 1s - loss: 1.9305 - accuracy: 0.3050 580/1563 [==========>...................] - ETA: 1s - loss: 1.9195 - accuracy: 0.3089 619/1563 [==========>...................] - ETA: 1s - loss: 1.9109 - accuracy: 0.3127 658/1563 [===========>..................] - ETA: 1s - loss: 1.9025 - accuracy: 0.3163 697/1563 [============>.................] - ETA: 1s - loss: 1.8957 - accuracy: 0.3198 736/1563 [=============>................] - ETA: 1s - loss: 1.8884 - accuracy: 0.3217 775/1563 [=============>................] - ETA: 1s - loss: 1.8835 - accuracy: 0.3241 814/1563 [==============>...............] - ETA: 0s - loss: 1.8757 - accuracy: 0.3281 853/1563 [===============>..............] - ETA: 0s - loss: 1.8726 - accuracy: 0.3296 892/1563 [================>.............] - ETA: 0s - loss: 1.8680 - accuracy: 0.3311 931/1563 [================>.............] - ETA: 0s - loss: 1.8643 - accuracy: 0.3324 970/1563 [=================>............] - ETA: 0s - loss: 1.8607 - accuracy: 0.3343 1009/1563 [==================>...........] - ETA: 0s - loss: 1.8563 - accuracy: 0.3357 1048/1563 [===================>..........] - ETA: 0s - loss: 1.8514 - accuracy: 0.3378 1087/1563 [===================>..........] - ETA: 0s - loss: 1.8475 - accuracy: 0.3392 1126/1563 [====================>.........] - ETA: 0s - loss: 1.8432 - accuracy: 0.3420 1165/1563 [=====================>........] - ETA: 0s - loss: 1.8405 - accuracy: 0.3434 1204/1563 [======================>.......] - ETA: 0s - loss: 1.8367 - accuracy: 0.3460 1243/1563 [======================>.......] - ETA: 0s - loss: 1.8330 - accuracy: 0.3470 1282/1563 [=======================>......] - ETA: 0s - loss: 1.8288 - accuracy: 0.3483 1321/1563 [========================>.....] - ETA: 0s - loss: 1.8246 - accuracy: 0.3499 1360/1563 [=========================>....] - ETA: 0s - loss: 1.8214 - accuracy: 0.3507 1399/1563 [=========================>....] - ETA: 0s - loss: 1.8179 - accuracy: 0.3518 1438/1563 [==========================>...] - ETA: 0s - loss: 1.8155 - accuracy: 0.3523 1477/1563 [===========================>..] - ETA: 0s - loss: 1.8126 - accuracy: 0.3533 1516/1563 [============================>.] - ETA: 0s - loss: 1.8091 - accuracy: 0.3550 1555/1563 [============================>.] - ETA: 0s - loss: 1.8064 - accuracy: 0.3563 1563/1563 [==============================] - 6s 1ms/step - loss: 1.8061 - accuracy: 0.3565 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 2.2452 - accuracy: 0.2812 40/1563 [..............................] - ETA: 1s - loss: 1.7232 - accuracy: 0.4062 78/1563 [>.............................] - ETA: 1s - loss: 1.6935 - accuracy: 0.4119 117/1563 [=>............................] - ETA: 1s - loss: 1.6750 - accuracy: 0.4129 156/1563 [=>............................] - ETA: 1s - loss: 1.6877 - accuracy: 0.4056 195/1563 [==>...........................] - ETA: 1s - loss: 1.6723 - accuracy: 0.4101 234/1563 [===>..........................] - ETA: 1s - loss: 1.6745 - accuracy: 0.4132 273/1563 [====>.........................] - ETA: 1s - loss: 1.6739 - accuracy: 0.4107 312/1563 [====>.........................] - ETA: 1s - loss: 1.6662 - accuracy: 0.4140 351/1563 [=====>........................] - ETA: 1s - loss: 1.6660 - accuracy: 0.4150 390/1563 [======>.......................] - ETA: 1s - loss: 1.6686 - accuracy: 0.4141 429/1563 [=======>......................] - ETA: 1s - loss: 1.6661 - accuracy: 0.4148 468/1563 [=======>......................] - ETA: 1s - loss: 1.6662 - accuracy: 0.4141 507/1563 [========>.....................] - ETA: 1s - loss: 1.6605 - accuracy: 0.4163 546/1563 [=========>....................] - ETA: 1s - loss: 1.6569 - accuracy: 0.4174 585/1563 [==========>...................] - ETA: 1s - loss: 1.6522 - accuracy: 0.4187 624/1563 [==========>...................] - ETA: 1s - loss: 1.6539 - accuracy: 0.4175 663/1563 [===========>..................] - ETA: 1s - loss: 1.6530 - accuracy: 0.4176 702/1563 [============>.................] - ETA: 1s - loss: 1.6495 - accuracy: 0.4192 741/1563 [=============>................] - ETA: 1s - loss: 1.6487 - accuracy: 0.4200 780/1563 [=============>................] - ETA: 1s - loss: 1.6468 - accuracy: 0.4211 819/1563 [==============>...............] - ETA: 0s - loss: 1.6436 - accuracy: 0.4227 857/1563 [===============>..............] - ETA: 0s - loss: 1.6441 - accuracy: 0.4223 895/1563 [================>.............] - ETA: 0s - loss: 1.6423 - accuracy: 0.4233 934/1563 [================>.............] - ETA: 0s - loss: 1.6421 - accuracy: 0.4239 972/1563 [=================>............] - ETA: 0s - loss: 1.6414 - accuracy: 0.4240 1010/1563 [==================>...........] - ETA: 0s - loss: 1.6397 - accuracy: 0.4245 1048/1563 [===================>..........] - ETA: 0s - loss: 1.6398 - accuracy: 0.4245 1087/1563 [===================>..........] - ETA: 0s - loss: 1.6382 - accuracy: 0.4249 1125/1563 [====================>.........] - ETA: 0s - loss: 1.6377 - accuracy: 0.4250 1164/1563 [=====================>........] - ETA: 0s - loss: 1.6373 - accuracy: 0.4250 1203/1563 [======================>.......] - ETA: 0s - loss: 1.6368 - accuracy: 0.4254 1242/1563 [======================>.......] - ETA: 0s - loss: 1.6353 - accuracy: 0.4258 1280/1563 [=======================>......] - ETA: 0s - loss: 1.6351 - accuracy: 0.4260 1318/1563 [========================>.....] - ETA: 0s - loss: 1.6349 - accuracy: 0.4258 1356/1563 [=========================>....] - ETA: 0s - loss: 1.6336 - accuracy: 0.4259 1394/1563 [=========================>....] - ETA: 0s - loss: 1.6318 - accuracy: 0.4263 1432/1563 [==========================>...] - ETA: 0s - loss: 1.6298 - accuracy: 0.4268 1471/1563 [===========================>..] - ETA: 0s - loss: 1.6285 - accuracy: 0.4274 1509/1563 [===========================>..] - ETA: 0s - loss: 1.6259 - accuracy: 0.4283 1548/1563 [============================>.] - ETA: 0s - loss: 1.6242 - accuracy: 0.4284 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6240 - accuracy: 0.4284 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.6127 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.5500 - accuracy: 0.4455 78/1563 [>.............................] - ETA: 1s - loss: 1.5528 - accuracy: 0.4403 117/1563 [=>............................] - ETA: 1s - loss: 1.5743 - accuracy: 0.4418 155/1563 [=>............................] - ETA: 1s - loss: 1.5796 - accuracy: 0.4405 194/1563 [==>...........................] - ETA: 1s - loss: 1.5803 - accuracy: 0.4398 232/1563 [===>..........................] - ETA: 1s - loss: 1.5691 - accuracy: 0.4463 270/1563 [====>.........................] - ETA: 1s - loss: 1.5624 - accuracy: 0.4486 308/1563 [====>.........................] - ETA: 1s - loss: 1.5620 - accuracy: 0.4500 346/1563 [=====>........................] - ETA: 1s - loss: 1.5579 - accuracy: 0.4530 385/1563 [======>.......................] - ETA: 1s - loss: 1.5564 - accuracy: 0.4532 424/1563 [=======>......................] - ETA: 1s - loss: 1.5605 - accuracy: 0.4501 463/1563 [=======>......................] - ETA: 1s - loss: 1.5628 - accuracy: 0.4486 502/1563 [========>.....................] - ETA: 1s - loss: 1.5608 - accuracy: 0.4498 541/1563 [=========>....................] - ETA: 1s - loss: 1.5569 - accuracy: 0.4503 580/1563 [==========>...................] - ETA: 1s - loss: 1.5558 - accuracy: 0.4503 619/1563 [==========>...................] - ETA: 1s - loss: 1.5552 - accuracy: 0.4514 657/1563 [===========>..................] - ETA: 1s - loss: 1.5546 - accuracy: 0.4519 695/1563 [============>.................] - ETA: 1s - loss: 1.5530 - accuracy: 0.4527 733/1563 [=============>................] - ETA: 1s - loss: 1.5535 - accuracy: 0.4528 771/1563 [=============>................] - ETA: 1s - loss: 1.5524 - accuracy: 0.4534 810/1563 [==============>...............] - ETA: 0s - loss: 1.5531 - accuracy: 0.4530 849/1563 [===============>..............] - ETA: 0s - loss: 1.5516 - accuracy: 0.4534 888/1563 [================>.............] - ETA: 0s - loss: 1.5538 - accuracy: 0.4528 926/1563 [================>.............] - ETA: 0s - loss: 1.5529 - accuracy: 0.4527 965/1563 [=================>............] - ETA: 0s - loss: 1.5515 - accuracy: 0.4533 1003/1563 [==================>...........] - ETA: 0s - loss: 1.5516 - accuracy: 0.4537 1041/1563 [==================>...........] - ETA: 0s - loss: 1.5508 - accuracy: 0.4539 1079/1563 [===================>..........] - ETA: 0s - loss: 1.5488 - accuracy: 0.4542 1117/1563 [====================>.........] - ETA: 0s - loss: 1.5475 - accuracy: 0.4549 1156/1563 [=====================>........] - ETA: 0s - loss: 1.5462 - accuracy: 0.4553 1195/1563 [=====================>........] - ETA: 0s - loss: 1.5450 - accuracy: 0.4548 1234/1563 [======================>.......] - ETA: 0s - loss: 1.5443 - accuracy: 0.4552 1273/1563 [=======================>......] - ETA: 0s - loss: 1.5446 - accuracy: 0.4551 1312/1563 [========================>.....] - ETA: 0s - loss: 1.5443 - accuracy: 0.4552 1349/1563 [========================>.....] - ETA: 0s - loss: 1.5431 - accuracy: 0.4558 1387/1563 [=========================>....] - ETA: 0s - loss: 1.5426 - accuracy: 0.4559 1426/1563 [==========================>...] - ETA: 0s - loss: 1.5426 - accuracy: 0.4564 1465/1563 [===========================>..] - ETA: 0s - loss: 1.5425 - accuracy: 0.4564 1503/1563 [===========================>..] - ETA: 0s - loss: 1.5425 - accuracy: 0.4568 1542/1563 [============================>.] - ETA: 0s - loss: 1.5416 - accuracy: 0.4573 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5422 - accuracy: 0.4571 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.6679 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.4869 - accuracy: 0.4744 77/1563 [>.............................] - ETA: 1s - loss: 1.5256 - accuracy: 0.4582 116/1563 [=>............................] - ETA: 1s - loss: 1.5075 - accuracy: 0.4671 155/1563 [=>............................] - ETA: 1s - loss: 1.4893 - accuracy: 0.4770 194/1563 [==>...........................] - ETA: 1s - loss: 1.4980 - accuracy: 0.4702 232/1563 [===>..........................] - ETA: 1s - loss: 1.4943 - accuracy: 0.4709 271/1563 [====>.........................] - ETA: 1s - loss: 1.4899 - accuracy: 0.4730 309/1563 [====>.........................] - ETA: 1s - loss: 1.4897 - accuracy: 0.4730 347/1563 [=====>........................] - ETA: 1s - loss: 1.4941 - accuracy: 0.4713 385/1563 [======>.......................] - ETA: 1s - loss: 1.4968 - accuracy: 0.4722 424/1563 [=======>......................] - ETA: 1s - loss: 1.4974 - accuracy: 0.4731 462/1563 [=======>......................] - ETA: 1s - loss: 1.4966 - accuracy: 0.4738 500/1563 [========>.....................] - ETA: 1s - loss: 1.4968 - accuracy: 0.4748 539/1563 [=========>....................] - ETA: 1s - loss: 1.4939 - accuracy: 0.4761 577/1563 [==========>...................] - ETA: 1s - loss: 1.4936 - accuracy: 0.4767 616/1563 [==========>...................] - ETA: 1s - loss: 1.4900 - accuracy: 0.4778 654/1563 [===========>..................] - ETA: 1s - loss: 1.4901 - accuracy: 0.4774 692/1563 [============>.................] - ETA: 1s - loss: 1.4893 - accuracy: 0.4773 730/1563 [=============>................] - ETA: 1s - loss: 1.4913 - accuracy: 0.4765 769/1563 [=============>................] - ETA: 1s - loss: 1.4911 - accuracy: 0.4764 808/1563 [==============>...............] - ETA: 0s - loss: 1.4887 - accuracy: 0.4776 847/1563 [===============>..............] - ETA: 0s - loss: 1.4861 - accuracy: 0.4781 886/1563 [================>.............] - ETA: 0s - loss: 1.4861 - accuracy: 0.4775 925/1563 [================>.............] - ETA: 0s - loss: 1.4855 - accuracy: 0.4773 963/1563 [=================>............] - ETA: 0s - loss: 1.4850 - accuracy: 0.4776 1001/1563 [==================>...........] - ETA: 0s - loss: 1.4834 - accuracy: 0.4786 1039/1563 [==================>...........] - ETA: 0s - loss: 1.4828 - accuracy: 0.4792 1078/1563 [===================>..........] - ETA: 0s - loss: 1.4823 - accuracy: 0.4791 1117/1563 [====================>.........] - ETA: 0s - loss: 1.4821 - accuracy: 0.4787 1156/1563 [=====================>........] - ETA: 0s - loss: 1.4827 - accuracy: 0.4782 1195/1563 [=====================>........] - ETA: 0s - loss: 1.4838 - accuracy: 0.4776 1233/1563 [======================>.......] - ETA: 0s - loss: 1.4827 - accuracy: 0.4781 1271/1563 [=======================>......] - ETA: 0s - loss: 1.4817 - accuracy: 0.4785 1309/1563 [========================>.....] - ETA: 0s - loss: 1.4823 - accuracy: 0.4788 1348/1563 [========================>.....] - ETA: 0s - loss: 1.4828 - accuracy: 0.4787 1387/1563 [=========================>....] - ETA: 0s - loss: 1.4839 - accuracy: 0.4785 1426/1563 [==========================>...] - ETA: 0s - loss: 1.4843 - accuracy: 0.4783 1465/1563 [===========================>..] - ETA: 0s - loss: 1.4841 - accuracy: 0.4780 1504/1563 [===========================>..] - ETA: 0s - loss: 1.4834 - accuracy: 0.4783 1542/1563 [============================>.] - ETA: 0s - loss: 1.4837 - accuracy: 0.4781 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4830 - accuracy: 0.4779 Epoch 5/10 1/1563 [..............................] - ETA: 1s - loss: 1.3014 - accuracy: 0.5625 39/1563 [..............................] - ETA: 2s - loss: 1.4907 - accuracy: 0.4776 78/1563 [>.............................] - ETA: 1s - loss: 1.4549 - accuracy: 0.4896 117/1563 [=>............................] - ETA: 1s - loss: 1.4510 - accuracy: 0.4861 156/1563 [=>............................] - ETA: 1s - loss: 1.4349 - accuracy: 0.4938 195/1563 [==>...........................] - ETA: 1s - loss: 1.4418 - accuracy: 0.4912 233/1563 [===>..........................] - ETA: 1s - loss: 1.4490 - accuracy: 0.4882 271/1563 [====>.........................] - ETA: 1s - loss: 1.4485 - accuracy: 0.4890 309/1563 [====>.........................] - ETA: 1s - loss: 1.4456 - accuracy: 0.4908 347/1563 [=====>........................] - ETA: 1s - loss: 1.4432 - accuracy: 0.4934 385/1563 [======>.......................] - ETA: 1s - loss: 1.4410 - accuracy: 0.4933 422/1563 [=======>......................] - ETA: 1s - loss: 1.4417 - accuracy: 0.4926 461/1563 [=======>......................] - ETA: 1s - loss: 1.4418 - accuracy: 0.4929 500/1563 [========>.....................] - ETA: 1s - loss: 1.4423 - accuracy: 0.4936 539/1563 [=========>....................] - ETA: 1s - loss: 1.4460 - accuracy: 0.4917 577/1563 [==========>...................] - ETA: 1s - loss: 1.4462 - accuracy: 0.4928 616/1563 [==========>...................] - ETA: 1s - loss: 1.4423 - accuracy: 0.4934 655/1563 [===========>..................] - ETA: 1s - loss: 1.4411 - accuracy: 0.4938 694/1563 [============>.................] - ETA: 1s - loss: 1.4437 - accuracy: 0.4933 733/1563 [=============>................] - ETA: 1s - loss: 1.4433 - accuracy: 0.4929 772/1563 [=============>................] - ETA: 1s - loss: 1.4439 - accuracy: 0.4921 811/1563 [==============>...............] - ETA: 0s - loss: 1.4407 - accuracy: 0.4933 850/1563 [===============>..............] - ETA: 0s - loss: 1.4379 - accuracy: 0.4937 889/1563 [================>.............] - ETA: 0s - loss: 1.4376 - accuracy: 0.4938 928/1563 [================>.............] - ETA: 0s - loss: 1.4392 - accuracy: 0.4927 967/1563 [=================>............] - ETA: 0s - loss: 1.4400 - accuracy: 0.4923 1006/1563 [==================>...........] - ETA: 0s - loss: 1.4393 - accuracy: 0.4932 1044/1563 [===================>..........] - ETA: 0s - loss: 1.4375 - accuracy: 0.4941 1083/1563 [===================>..........] - ETA: 0s - loss: 1.4384 - accuracy: 0.4942 1122/1563 [====================>.........] - ETA: 0s - loss: 1.4381 - accuracy: 0.4948 1161/1563 [=====================>........] - ETA: 0s - loss: 1.4386 - accuracy: 0.4946 1200/1563 [======================>.......] - ETA: 0s - loss: 1.4383 - accuracy: 0.4947 1239/1563 [======================>.......] - ETA: 0s - loss: 1.4385 - accuracy: 0.4947 1278/1563 [=======================>......] - ETA: 0s - loss: 1.4390 - accuracy: 0.4945 1317/1563 [========================>.....] - ETA: 0s - loss: 1.4380 - accuracy: 0.4946 1355/1563 [=========================>....] - ETA: 0s - loss: 1.4380 - accuracy: 0.4945 1393/1563 [=========================>....] - ETA: 0s - loss: 1.4379 - accuracy: 0.4948 1432/1563 [==========================>...] - ETA: 0s - loss: 1.4382 - accuracy: 0.4946 1470/1563 [===========================>..] - ETA: 0s - loss: 1.4374 - accuracy: 0.4946 1509/1563 [===========================>..] - ETA: 0s - loss: 1.4355 - accuracy: 0.4951 1547/1563 [============================>.] - ETA: 0s - loss: 1.4341 - accuracy: 0.4952 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4334 - accuracy: 0.4954 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.3326 - accuracy: 0.5625 39/1563 [..............................] - ETA: 2s - loss: 1.4283 - accuracy: 0.4880 78/1563 [>.............................] - ETA: 1s - loss: 1.3962 - accuracy: 0.5024 117/1563 [=>............................] - ETA: 1s - loss: 1.4035 - accuracy: 0.5021 156/1563 [=>............................] - ETA: 1s - loss: 1.3971 - accuracy: 0.5066 195/1563 [==>...........................] - ETA: 1s - loss: 1.3905 - accuracy: 0.5111 234/1563 [===>..........................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5097 273/1563 [====>.........................] - ETA: 1s - loss: 1.3935 - accuracy: 0.5127 312/1563 [====>.........................] - ETA: 1s - loss: 1.3901 - accuracy: 0.5133 351/1563 [=====>........................] - ETA: 1s - loss: 1.3884 - accuracy: 0.5148 390/1563 [======>.......................] - ETA: 1s - loss: 1.3891 - accuracy: 0.5154 429/1563 [=======>......................] - ETA: 1s - loss: 1.3880 - accuracy: 0.5152 468/1563 [=======>......................] - ETA: 1s - loss: 1.3913 - accuracy: 0.5142 506/1563 [========>.....................] - ETA: 1s - loss: 1.3940 - accuracy: 0.5134 544/1563 [=========>....................] - ETA: 1s - loss: 1.3933 - accuracy: 0.5124 583/1563 [==========>...................] - ETA: 1s - loss: 1.3914 - accuracy: 0.5133 621/1563 [==========>...................] - ETA: 1s - loss: 1.3930 - accuracy: 0.5125 659/1563 [===========>..................] - ETA: 1s - loss: 1.3936 - accuracy: 0.5126 697/1563 [============>.................] - ETA: 1s - loss: 1.3942 - accuracy: 0.5121 735/1563 [=============>................] - ETA: 1s - loss: 1.3929 - accuracy: 0.5123 773/1563 [=============>................] - ETA: 1s - loss: 1.3932 - accuracy: 0.5122 812/1563 [==============>...............] - ETA: 0s - loss: 1.3938 - accuracy: 0.5120 851/1563 [===============>..............] - ETA: 0s - loss: 1.3933 - accuracy: 0.5118 890/1563 [================>.............] - ETA: 0s - loss: 1.3944 - accuracy: 0.5110 929/1563 [================>.............] - ETA: 0s - loss: 1.3918 - accuracy: 0.5125 968/1563 [=================>............] - ETA: 0s - loss: 1.3912 - accuracy: 0.5126 1006/1563 [==================>...........] - ETA: 0s - loss: 1.3889 - accuracy: 0.5128 1045/1563 [===================>..........] - ETA: 0s - loss: 1.3895 - accuracy: 0.5124 1084/1563 [===================>..........] - ETA: 0s - loss: 1.3884 - accuracy: 0.5128 1123/1563 [====================>.........] - ETA: 0s - loss: 1.3872 - accuracy: 0.5128 1162/1563 [=====================>........] - ETA: 0s - loss: 1.3871 - accuracy: 0.5126 1201/1563 [======================>.......] - ETA: 0s - loss: 1.3876 - accuracy: 0.5129 1239/1563 [======================>.......] - ETA: 0s - loss: 1.3862 - accuracy: 0.5136 1277/1563 [=======================>......] - ETA: 0s - loss: 1.3868 - accuracy: 0.5134 1316/1563 [========================>.....] - ETA: 0s - loss: 1.3870 - accuracy: 0.5132 1354/1563 [========================>.....] - ETA: 0s - loss: 1.3877 - accuracy: 0.5127 1393/1563 [=========================>....] - ETA: 0s - loss: 1.3871 - accuracy: 0.5126 1432/1563 [==========================>...] - ETA: 0s - loss: 1.3886 - accuracy: 0.5120 1471/1563 [===========================>..] - ETA: 0s - loss: 1.3890 - accuracy: 0.5121 1509/1563 [===========================>..] - ETA: 0s - loss: 1.3895 - accuracy: 0.5120 1548/1563 [============================>.] - ETA: 0s - loss: 1.3885 - accuracy: 0.5127 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3890 - accuracy: 0.5124 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.3187 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.3758 - accuracy: 0.5192 77/1563 [>.............................] - ETA: 1s - loss: 1.3632 - accuracy: 0.5284 116/1563 [=>............................] - ETA: 1s - loss: 1.3668 - accuracy: 0.5242 155/1563 [=>............................] - ETA: 1s - loss: 1.3652 - accuracy: 0.5244 194/1563 [==>...........................] - ETA: 1s - loss: 1.3650 - accuracy: 0.5240 233/1563 [===>..........................] - ETA: 1s - loss: 1.3592 - accuracy: 0.5264 271/1563 [====>.........................] - ETA: 1s - loss: 1.3573 - accuracy: 0.5261 309/1563 [====>.........................] - ETA: 1s - loss: 1.3562 - accuracy: 0.5254 348/1563 [=====>........................] - ETA: 1s - loss: 1.3537 - accuracy: 0.5265 387/1563 [======>.......................] - ETA: 1s - loss: 1.3507 - accuracy: 0.5265 426/1563 [=======>......................] - ETA: 1s - loss: 1.3527 - accuracy: 0.5259 465/1563 [=======>......................] - ETA: 1s - loss: 1.3536 - accuracy: 0.5261 504/1563 [========>.....................] - ETA: 1s - loss: 1.3519 - accuracy: 0.5276 543/1563 [=========>....................] - ETA: 1s - loss: 1.3512 - accuracy: 0.5275 582/1563 [==========>...................] - ETA: 1s - loss: 1.3518 - accuracy: 0.5270 621/1563 [==========>...................] - ETA: 1s - loss: 1.3537 - accuracy: 0.5256 660/1563 [===========>..................] - ETA: 1s - loss: 1.3558 - accuracy: 0.5248 699/1563 [============>.................] - ETA: 1s - loss: 1.3521 - accuracy: 0.5254 738/1563 [=============>................] - ETA: 1s - loss: 1.3554 - accuracy: 0.5241 777/1563 [=============>................] - ETA: 1s - loss: 1.3550 - accuracy: 0.5247 816/1563 [==============>...............] - ETA: 0s - loss: 1.3571 - accuracy: 0.5239 855/1563 [===============>..............] - ETA: 0s - loss: 1.3574 - accuracy: 0.5246 893/1563 [================>.............] - ETA: 0s - loss: 1.3592 - accuracy: 0.5238 931/1563 [================>.............] - ETA: 0s - loss: 1.3584 - accuracy: 0.5244 970/1563 [=================>............] - ETA: 0s - loss: 1.3553 - accuracy: 0.5246 1009/1563 [==================>...........] - ETA: 0s - loss: 1.3552 - accuracy: 0.5249 1048/1563 [===================>..........] - ETA: 0s - loss: 1.3526 - accuracy: 0.5257 1087/1563 [===================>..........] - ETA: 0s - loss: 1.3521 - accuracy: 0.5259 1126/1563 [====================>.........] - ETA: 0s - loss: 1.3525 - accuracy: 0.5254 1165/1563 [=====================>........] - ETA: 0s - loss: 1.3525 - accuracy: 0.5252 1204/1563 [======================>.......] - ETA: 0s - loss: 1.3522 - accuracy: 0.5249 1243/1563 [======================>.......] - ETA: 0s - loss: 1.3523 - accuracy: 0.5251 1282/1563 [=======================>......] - ETA: 0s - loss: 1.3519 - accuracy: 0.5255 1321/1563 [========================>.....] - ETA: 0s - loss: 1.3518 - accuracy: 0.5256 1360/1563 [=========================>....] - ETA: 0s - loss: 1.3514 - accuracy: 0.5256 1399/1563 [=========================>....] - ETA: 0s - loss: 1.3515 - accuracy: 0.5258 1438/1563 [==========================>...] - ETA: 0s - loss: 1.3513 - accuracy: 0.5264 1477/1563 [===========================>..] - ETA: 0s - loss: 1.3506 - accuracy: 0.5270 1516/1563 [============================>.] - ETA: 0s - loss: 1.3499 - accuracy: 0.5271 1555/1563 [============================>.] - ETA: 0s - loss: 1.3507 - accuracy: 0.5268 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3513 - accuracy: 0.5266 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.4238 - accuracy: 0.5000 39/1563 [..............................] - ETA: 2s - loss: 1.2738 - accuracy: 0.5689 78/1563 [>.............................] - ETA: 1s - loss: 1.2916 - accuracy: 0.5505 117/1563 [=>............................] - ETA: 1s - loss: 1.3239 - accuracy: 0.5358 156/1563 [=>............................] - ETA: 1s - loss: 1.3218 - accuracy: 0.5371 195/1563 [==>...........................] - ETA: 1s - loss: 1.3234 - accuracy: 0.5362 234/1563 [===>..........................] - ETA: 1s - loss: 1.3208 - accuracy: 0.5371 273/1563 [====>.........................] - ETA: 1s - loss: 1.3169 - accuracy: 0.5394 312/1563 [====>.........................] - ETA: 1s - loss: 1.3177 - accuracy: 0.5395 351/1563 [=====>........................] - ETA: 1s - loss: 1.3127 - accuracy: 0.5424 390/1563 [======>.......................] - ETA: 1s - loss: 1.3073 - accuracy: 0.5441 429/1563 [=======>......................] - ETA: 1s - loss: 1.3124 - accuracy: 0.5415 468/1563 [=======>......................] - ETA: 1s - loss: 1.3157 - accuracy: 0.5393 507/1563 [========>.....................] - ETA: 1s - loss: 1.3179 - accuracy: 0.5378 546/1563 [=========>....................] - ETA: 1s - loss: 1.3172 - accuracy: 0.5374 584/1563 [==========>...................] - ETA: 1s - loss: 1.3155 - accuracy: 0.5390 623/1563 [==========>...................] - ETA: 1s - loss: 1.3166 - accuracy: 0.5393 662/1563 [===========>..................] - ETA: 1s - loss: 1.3197 - accuracy: 0.5382 701/1563 [============>.................] - ETA: 1s - loss: 1.3191 - accuracy: 0.5388 740/1563 [=============>................] - ETA: 1s - loss: 1.3215 - accuracy: 0.5389 779/1563 [=============>................] - ETA: 1s - loss: 1.3203 - accuracy: 0.5393 818/1563 [==============>...............] - ETA: 0s - loss: 1.3184 - accuracy: 0.5399 857/1563 [===============>..............] - ETA: 0s - loss: 1.3154 - accuracy: 0.5411 896/1563 [================>.............] - ETA: 0s - loss: 1.3177 - accuracy: 0.5399 935/1563 [================>.............] - ETA: 0s - loss: 1.3170 - accuracy: 0.5402 974/1563 [=================>............] - ETA: 0s - loss: 1.3150 - accuracy: 0.5409 1013/1563 [==================>...........] - ETA: 0s - loss: 1.3162 - accuracy: 0.5412 1052/1563 [===================>..........] - ETA: 0s - loss: 1.3148 - accuracy: 0.5417 1091/1563 [===================>..........] - ETA: 0s - loss: 1.3150 - accuracy: 0.5416 1130/1563 [====================>.........] - ETA: 0s - loss: 1.3164 - accuracy: 0.5412 1168/1563 [=====================>........] - ETA: 0s - loss: 1.3162 - accuracy: 0.5407 1206/1563 [======================>.......] - ETA: 0s - loss: 1.3161 - accuracy: 0.5406 1245/1563 [======================>.......] - ETA: 0s - loss: 1.3174 - accuracy: 0.5401 1284/1563 [=======================>......] - ETA: 0s - loss: 1.3181 - accuracy: 0.5398 1323/1563 [========================>.....] - ETA: 0s - loss: 1.3190 - accuracy: 0.5394 1362/1563 [=========================>....] - ETA: 0s - loss: 1.3185 - accuracy: 0.5393 1401/1563 [=========================>....] - ETA: 0s - loss: 1.3182 - accuracy: 0.5392 1440/1563 [==========================>...] - ETA: 0s - loss: 1.3183 - accuracy: 0.5387 1478/1563 [===========================>..] - ETA: 0s - loss: 1.3173 - accuracy: 0.5386 1516/1563 [============================>.] - ETA: 0s - loss: 1.3174 - accuracy: 0.5380 1555/1563 [============================>.] - ETA: 0s - loss: 1.3162 - accuracy: 0.5381 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3163 - accuracy: 0.5382 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.2726 - accuracy: 0.5938 40/1563 [..............................] - ETA: 2s - loss: 1.2463 - accuracy: 0.5656 79/1563 [>.............................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5578 118/1563 [=>............................] - ETA: 1s - loss: 1.2510 - accuracy: 0.5644 156/1563 [=>............................] - ETA: 1s - loss: 1.2569 - accuracy: 0.5639 195/1563 [==>...........................] - ETA: 1s - loss: 1.2572 - accuracy: 0.5611 234/1563 [===>..........................] - ETA: 1s - loss: 1.2549 - accuracy: 0.5621 273/1563 [====>.........................] - ETA: 1s - loss: 1.2545 - accuracy: 0.5619 312/1563 [====>.........................] - ETA: 1s - loss: 1.2644 - accuracy: 0.5577 351/1563 [=====>........................] - ETA: 1s - loss: 1.2678 - accuracy: 0.5572 390/1563 [======>.......................] - ETA: 1s - loss: 1.2692 - accuracy: 0.5553 429/1563 [=======>......................] - ETA: 1s - loss: 1.2699 - accuracy: 0.5558 467/1563 [=======>......................] - ETA: 1s - loss: 1.2746 - accuracy: 0.5537 506/1563 [========>.....................] - ETA: 1s - loss: 1.2761 - accuracy: 0.5548 545/1563 [=========>....................] - ETA: 1s - loss: 1.2752 - accuracy: 0.5548 584/1563 [==========>...................] - ETA: 1s - loss: 1.2770 - accuracy: 0.5539 622/1563 [==========>...................] - ETA: 1s - loss: 1.2771 - accuracy: 0.5539 661/1563 [===========>..................] - ETA: 1s - loss: 1.2766 - accuracy: 0.5536 700/1563 [============>.................] - ETA: 1s - loss: 1.2770 - accuracy: 0.5533 739/1563 [=============>................] - ETA: 1s - loss: 1.2775 - accuracy: 0.5523 778/1563 [=============>................] - ETA: 1s - loss: 1.2786 - accuracy: 0.5511 817/1563 [==============>...............] - ETA: 0s - loss: 1.2790 - accuracy: 0.5505 855/1563 [===============>..............] - ETA: 0s - loss: 1.2790 - accuracy: 0.5503 893/1563 [================>.............] - ETA: 0s - loss: 1.2796 - accuracy: 0.5511 932/1563 [================>.............] - ETA: 0s - loss: 1.2799 - accuracy: 0.5510 971/1563 [=================>............] - ETA: 0s - loss: 1.2797 - accuracy: 0.5517 1010/1563 [==================>...........] - ETA: 0s - loss: 1.2823 - accuracy: 0.5511 1049/1563 [===================>..........] - ETA: 0s - loss: 1.2820 - accuracy: 0.5517 1087/1563 [===================>..........] - ETA: 0s - loss: 1.2823 - accuracy: 0.5513 1126/1563 [====================>.........] - ETA: 0s - loss: 1.2827 - accuracy: 0.5509 1165/1563 [=====================>........] - ETA: 0s - loss: 1.2829 - accuracy: 0.5505 1203/1563 [======================>.......] - ETA: 0s - loss: 1.2823 - accuracy: 0.5508 1242/1563 [======================>.......] - ETA: 0s - loss: 1.2840 - accuracy: 0.5503 1281/1563 [=======================>......] - ETA: 0s - loss: 1.2832 - accuracy: 0.5506 1320/1563 [========================>.....] - ETA: 0s - loss: 1.2838 - accuracy: 0.5509 1358/1563 [=========================>....] - ETA: 0s - loss: 1.2827 - accuracy: 0.5516 1397/1563 [=========================>....] - ETA: 0s - loss: 1.2832 - accuracy: 0.5510 1436/1563 [==========================>...] - ETA: 0s - loss: 1.2837 - accuracy: 0.5508 1474/1563 [===========================>..] - ETA: 0s - loss: 1.2854 - accuracy: 0.5497 1512/1563 [============================>.] - ETA: 0s - loss: 1.2860 - accuracy: 0.5493 1551/1563 [============================>.] - ETA: 0s - loss: 1.2849 - accuracy: 0.5499 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2848 - accuracy: 0.5501 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 0.9229 - accuracy: 0.7188 40/1563 [..............................] - ETA: 1s - loss: 1.2499 - accuracy: 0.5656 79/1563 [>.............................] - ETA: 1s - loss: 1.2363 - accuracy: 0.5696 117/1563 [=>............................] - ETA: 1s - loss: 1.2323 - accuracy: 0.5697 155/1563 [=>............................] - ETA: 1s - loss: 1.2298 - accuracy: 0.5710 193/1563 [==>...........................] - ETA: 1s - loss: 1.2425 - accuracy: 0.5693 232/1563 [===>..........................] - ETA: 1s - loss: 1.2501 - accuracy: 0.5657 271/1563 [====>.........................] - ETA: 1s - loss: 1.2470 - accuracy: 0.5638 310/1563 [====>.........................] - ETA: 1s - loss: 1.2494 - accuracy: 0.5633 348/1563 [=====>........................] - ETA: 1s - loss: 1.2498 - accuracy: 0.5629 387/1563 [======>.......................] - ETA: 1s - loss: 1.2468 - accuracy: 0.5639 426/1563 [=======>......................] - ETA: 1s - loss: 1.2468 - accuracy: 0.5659 464/1563 [=======>......................] - ETA: 1s - loss: 1.2497 - accuracy: 0.5644 503/1563 [========>.....................] - ETA: 1s - loss: 1.2482 - accuracy: 0.5655 541/1563 [=========>....................] - ETA: 1s - loss: 1.2513 - accuracy: 0.5641 579/1563 [==========>...................] - ETA: 1s - loss: 1.2506 - accuracy: 0.5647 618/1563 [==========>...................] - ETA: 1s - loss: 1.2514 - accuracy: 0.5645 657/1563 [===========>..................] - ETA: 1s - loss: 1.2541 - accuracy: 0.5628 696/1563 [============>.................] - ETA: 1s - loss: 1.2531 - accuracy: 0.5638 734/1563 [=============>................] - ETA: 1s - loss: 1.2542 - accuracy: 0.5624 773/1563 [=============>................] - ETA: 1s - loss: 1.2534 - accuracy: 0.5622 812/1563 [==============>...............] - ETA: 0s - loss: 1.2531 - accuracy: 0.5628 851/1563 [===============>..............] - ETA: 0s - loss: 1.2550 - accuracy: 0.5622 890/1563 [================>.............] - ETA: 0s - loss: 1.2524 - accuracy: 0.5624 928/1563 [================>.............] - ETA: 0s - loss: 1.2511 - accuracy: 0.5631 967/1563 [=================>............] - ETA: 0s - loss: 1.2523 - accuracy: 0.5623 1006/1563 [==================>...........] - ETA: 0s - loss: 1.2525 - accuracy: 0.5627 1044/1563 [===================>..........] - ETA: 0s - loss: 1.2543 - accuracy: 0.5619 1083/1563 [===================>..........] - ETA: 0s - loss: 1.2540 - accuracy: 0.5624 1121/1563 [====================>.........] - ETA: 0s - loss: 1.2512 - accuracy: 0.5632 1160/1563 [=====================>........] - ETA: 0s - loss: 1.2526 - accuracy: 0.5628 1199/1563 [======================>.......] - ETA: 0s - loss: 1.2535 - accuracy: 0.5624 1237/1563 [======================>.......] - ETA: 0s - loss: 1.2530 - accuracy: 0.5624 1276/1563 [=======================>......] - ETA: 0s - loss: 1.2530 - accuracy: 0.5625 1314/1563 [========================>.....] - ETA: 0s - loss: 1.2536 - accuracy: 0.5621 1353/1563 [========================>.....] - ETA: 0s - loss: 1.2532 - accuracy: 0.5624 1392/1563 [=========================>....] - ETA: 0s - loss: 1.2533 - accuracy: 0.5631 1431/1563 [==========================>...] - ETA: 0s - loss: 1.2538 - accuracy: 0.5630 1470/1563 [===========================>..] - ETA: 0s - loss: 1.2548 - accuracy: 0.5629 1509/1563 [===========================>..] - ETA: 0s - loss: 1.2541 - accuracy: 0.5635 1548/1563 [============================>.] - ETA: 0s - loss: 1.2543 - accuracy: 0.5629 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2543 - accuracy: 0.5628 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 36.35249137878418 hydro04 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins8731355674684206994.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS