Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 96143 queued and waiting for resources srun: job 96143 has been allocated resources 2024-10-31 07:43:07.658130: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-31 07:43:32.884883: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-10-31 07:44:06.006954: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-10-31 07:44:06.007582: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-10-31 07:44:06.007590: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-10-31 07:44:58.224922: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-31 07:45:01.244938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-10-31 07:45:08.210798: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:40:19 - loss: 2.4201 - accuracy: 0.1250 37/1563 [..............................] - ETA: 2s - loss: 2.2345 - accuracy: 0.1689 74/1563 [>.............................] - ETA: 2s - loss: 2.1828 - accuracy: 0.1964 112/1563 [=>............................] - ETA: 1s - loss: 2.1350 - accuracy: 0.2201 150/1563 [=>............................] - ETA: 1s - loss: 2.0990 - accuracy: 0.2352 188/1563 [==>...........................] - ETA: 1s - loss: 2.0762 - accuracy: 0.2497 226/1563 [===>..........................] - ETA: 1s - loss: 2.0525 - accuracy: 0.2618 264/1563 [====>.........................] - ETA: 1s - loss: 2.0328 - accuracy: 0.2685 302/1563 [====>.........................] - ETA: 1s - loss: 2.0084 - accuracy: 0.2791 340/1563 [=====>........................] - ETA: 1s - loss: 1.9932 - accuracy: 0.2837 378/1563 [======>.......................] - ETA: 1s - loss: 1.9833 - accuracy: 0.2884 416/1563 [======>.......................] - ETA: 1s - loss: 1.9720 - accuracy: 0.2927 453/1563 [=======>......................] - ETA: 1s - loss: 1.9632 - accuracy: 0.2961 490/1563 [========>.....................] - ETA: 1s - loss: 1.9522 - accuracy: 0.3013 527/1563 [=========>....................] - ETA: 1s - loss: 1.9455 - accuracy: 0.3029 565/1563 [=========>....................] - ETA: 1s - loss: 1.9375 - accuracy: 0.3055 603/1563 [==========>...................] - ETA: 1s - loss: 1.9328 - accuracy: 0.3080 641/1563 [===========>..................] - ETA: 1s - loss: 1.9251 - accuracy: 0.3113 678/1563 [============>.................] - ETA: 1s - loss: 1.9149 - accuracy: 0.3157 716/1563 [============>.................] - ETA: 1s - loss: 1.9063 - accuracy: 0.3188 754/1563 [=============>................] - ETA: 1s - loss: 1.9004 - accuracy: 0.3207 792/1563 [==============>...............] - ETA: 1s - loss: 1.8957 - accuracy: 0.3227 829/1563 [==============>...............] - ETA: 0s - loss: 1.8921 - accuracy: 0.3245 866/1563 [===============>..............] - ETA: 0s - loss: 1.8860 - accuracy: 0.3270 903/1563 [================>.............] - ETA: 0s - loss: 1.8797 - accuracy: 0.3292 940/1563 [=================>............] - ETA: 0s - loss: 1.8751 - accuracy: 0.3315 978/1563 [=================>............] - ETA: 0s - loss: 1.8696 - accuracy: 0.3333 1016/1563 [==================>...........] - ETA: 0s - loss: 1.8639 - accuracy: 0.3356 1054/1563 [===================>..........] - ETA: 0s - loss: 1.8600 - accuracy: 0.3371 1092/1563 [===================>..........] - ETA: 0s - loss: 1.8539 - accuracy: 0.3401 1130/1563 [====================>.........] - ETA: 0s - loss: 1.8513 - accuracy: 0.3410 1168/1563 [=====================>........] - ETA: 0s - loss: 1.8452 - accuracy: 0.3429 1206/1563 [======================>.......] - ETA: 0s - loss: 1.8411 - accuracy: 0.3447 1244/1563 [======================>.......] - ETA: 0s - loss: 1.8371 - accuracy: 0.3462 1282/1563 [=======================>......] - ETA: 0s - loss: 1.8357 - accuracy: 0.3467 1320/1563 [========================>.....] - ETA: 0s - loss: 1.8321 - accuracy: 0.3480 1358/1563 [=========================>....] - ETA: 0s - loss: 1.8301 - accuracy: 0.3484 1396/1563 [=========================>....] - ETA: 0s - loss: 1.8258 - accuracy: 0.3502 1434/1563 [==========================>...] - ETA: 0s - loss: 1.8220 - accuracy: 0.3513 1472/1563 [===========================>..] - ETA: 0s - loss: 1.8174 - accuracy: 0.3533 1510/1563 [===========================>..] - ETA: 0s - loss: 1.8140 - accuracy: 0.3542 1548/1563 [============================>.] - ETA: 0s - loss: 1.8114 - accuracy: 0.3552 1563/1563 [==============================] - 6s 1ms/step - loss: 1.8107 - accuracy: 0.3555 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.5013 - accuracy: 0.4062 38/1563 [..............................] - ETA: 2s - loss: 1.6619 - accuracy: 0.4186 76/1563 [>.............................] - ETA: 2s - loss: 1.6602 - accuracy: 0.4178 114/1563 [=>............................] - ETA: 1s - loss: 1.6563 - accuracy: 0.4208 152/1563 [=>............................] - ETA: 1s - loss: 1.6625 - accuracy: 0.4188 190/1563 [==>...........................] - ETA: 1s - loss: 1.6642 - accuracy: 0.4163 228/1563 [===>..........................] - ETA: 1s - loss: 1.6577 - accuracy: 0.4171 266/1563 [====>.........................] - ETA: 1s - loss: 1.6546 - accuracy: 0.4194 304/1563 [====>.........................] - ETA: 1s - loss: 1.6523 - accuracy: 0.4184 342/1563 [=====>........................] - ETA: 1s - loss: 1.6520 - accuracy: 0.4174 380/1563 [======>.......................] - ETA: 1s - loss: 1.6535 - accuracy: 0.4174 418/1563 [=======>......................] - ETA: 1s - loss: 1.6541 - accuracy: 0.4172 456/1563 [=======>......................] - ETA: 1s - loss: 1.6570 - accuracy: 0.4161 494/1563 [========>.....................] - ETA: 1s - loss: 1.6565 - accuracy: 0.4161 532/1563 [=========>....................] - ETA: 1s - loss: 1.6545 - accuracy: 0.4175 570/1563 [=========>....................] - ETA: 1s - loss: 1.6499 - accuracy: 0.4189 607/1563 [==========>...................] - ETA: 1s - loss: 1.6481 - accuracy: 0.4195 645/1563 [===========>..................] - ETA: 1s - loss: 1.6462 - accuracy: 0.4195 683/1563 [============>.................] - ETA: 1s - loss: 1.6453 - accuracy: 0.4194 721/1563 [============>.................] - ETA: 1s - loss: 1.6441 - accuracy: 0.4196 759/1563 [=============>................] - ETA: 1s - loss: 1.6415 - accuracy: 0.4217 797/1563 [==============>...............] - ETA: 1s - loss: 1.6415 - accuracy: 0.4217 835/1563 [===============>..............] - ETA: 0s - loss: 1.6414 - accuracy: 0.4222 873/1563 [===============>..............] - ETA: 0s - loss: 1.6413 - accuracy: 0.4217 911/1563 [================>.............] - ETA: 0s - loss: 1.6418 - accuracy: 0.4215 949/1563 [=================>............] - ETA: 0s - loss: 1.6399 - accuracy: 0.4222 986/1563 [=================>............] - ETA: 0s - loss: 1.6375 - accuracy: 0.4228 1024/1563 [==================>...........] - ETA: 0s - loss: 1.6369 - accuracy: 0.4229 1061/1563 [===================>..........] - ETA: 0s - loss: 1.6343 - accuracy: 0.4234 1099/1563 [====================>.........] - ETA: 0s - loss: 1.6344 - accuracy: 0.4230 1137/1563 [====================>.........] - ETA: 0s - loss: 1.6315 - accuracy: 0.4243 1175/1563 [=====================>........] - ETA: 0s - loss: 1.6304 - accuracy: 0.4247 1213/1563 [======================>.......] - ETA: 0s - loss: 1.6308 - accuracy: 0.4249 1251/1563 [=======================>......] - ETA: 0s - loss: 1.6300 - accuracy: 0.4253 1289/1563 [=======================>......] - ETA: 0s - loss: 1.6290 - accuracy: 0.4254 1327/1563 [========================>.....] - ETA: 0s - loss: 1.6274 - accuracy: 0.4257 1364/1563 [=========================>....] - ETA: 0s - loss: 1.6273 - accuracy: 0.4254 1401/1563 [=========================>....] - ETA: 0s - loss: 1.6262 - accuracy: 0.4258 1438/1563 [==========================>...] - ETA: 0s - loss: 1.6243 - accuracy: 0.4264 1476/1563 [===========================>..] - ETA: 0s - loss: 1.6236 - accuracy: 0.4267 1514/1563 [============================>.] - ETA: 0s - loss: 1.6230 - accuracy: 0.4271 1552/1563 [============================>.] - ETA: 0s - loss: 1.6214 - accuracy: 0.4275 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6210 - accuracy: 0.4278 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.8090 - accuracy: 0.2812 39/1563 [..............................] - ETA: 2s - loss: 1.5751 - accuracy: 0.4383 77/1563 [>.............................] - ETA: 1s - loss: 1.5768 - accuracy: 0.4501 115/1563 [=>............................] - ETA: 1s - loss: 1.5629 - accuracy: 0.4527 153/1563 [=>............................] - ETA: 1s - loss: 1.5600 - accuracy: 0.4471 191/1563 [==>...........................] - ETA: 1s - loss: 1.5536 - accuracy: 0.4504 228/1563 [===>..........................] - ETA: 1s - loss: 1.5508 - accuracy: 0.4527 266/1563 [====>.........................] - ETA: 1s - loss: 1.5553 - accuracy: 0.4474 303/1563 [====>.........................] - ETA: 1s - loss: 1.5545 - accuracy: 0.4483 340/1563 [=====>........................] - ETA: 1s - loss: 1.5579 - accuracy: 0.4476 378/1563 [======>.......................] - ETA: 1s - loss: 1.5541 - accuracy: 0.4489 416/1563 [======>.......................] - ETA: 1s - loss: 1.5559 - accuracy: 0.4512 454/1563 [=======>......................] - ETA: 1s - loss: 1.5576 - accuracy: 0.4496 491/1563 [========>.....................] - ETA: 1s - loss: 1.5601 - accuracy: 0.4515 527/1563 [=========>....................] - ETA: 1s - loss: 1.5594 - accuracy: 0.4529 565/1563 [=========>....................] - ETA: 1s - loss: 1.5587 - accuracy: 0.4539 603/1563 [==========>...................] - ETA: 1s - loss: 1.5594 - accuracy: 0.4533 641/1563 [===========>..................] - ETA: 1s - loss: 1.5605 - accuracy: 0.4512 679/1563 [============>.................] - ETA: 1s - loss: 1.5607 - accuracy: 0.4514 717/1563 [============>.................] - ETA: 1s - loss: 1.5561 - accuracy: 0.4526 755/1563 [=============>................] - ETA: 1s - loss: 1.5568 - accuracy: 0.4524 793/1563 [==============>...............] - ETA: 1s - loss: 1.5552 - accuracy: 0.4532 831/1563 [==============>...............] - ETA: 0s - loss: 1.5539 - accuracy: 0.4538 869/1563 [===============>..............] - ETA: 0s - loss: 1.5528 - accuracy: 0.4544 907/1563 [================>.............] - ETA: 0s - loss: 1.5517 - accuracy: 0.4552 945/1563 [=================>............] - ETA: 0s - loss: 1.5483 - accuracy: 0.4561 983/1563 [=================>............] - ETA: 0s - loss: 1.5462 - accuracy: 0.4565 1020/1563 [==================>...........] - ETA: 0s - loss: 1.5455 - accuracy: 0.4559 1058/1563 [===================>..........] - ETA: 0s - loss: 1.5450 - accuracy: 0.4557 1096/1563 [====================>.........] - ETA: 0s - loss: 1.5448 - accuracy: 0.4561 1134/1563 [====================>.........] - ETA: 0s - loss: 1.5462 - accuracy: 0.4554 1172/1563 [=====================>........] - ETA: 0s - loss: 1.5474 - accuracy: 0.4555 1210/1563 [======================>.......] - ETA: 0s - loss: 1.5463 - accuracy: 0.4556 1248/1563 [======================>.......] - ETA: 0s - loss: 1.5453 - accuracy: 0.4561 1286/1563 [=======================>......] - ETA: 0s - loss: 1.5460 - accuracy: 0.4556 1324/1563 [========================>.....] - ETA: 0s - loss: 1.5451 - accuracy: 0.4556 1361/1563 [=========================>....] - ETA: 0s - loss: 1.5456 - accuracy: 0.4556 1399/1563 [=========================>....] - ETA: 0s - loss: 1.5462 - accuracy: 0.4554 1437/1563 [==========================>...] - ETA: 0s - loss: 1.5452 - accuracy: 0.4561 1475/1563 [===========================>..] - ETA: 0s - loss: 1.5443 - accuracy: 0.4571 1513/1563 [============================>.] - ETA: 0s - loss: 1.5426 - accuracy: 0.4579 1551/1563 [============================>.] - ETA: 0s - loss: 1.5410 - accuracy: 0.4583 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5409 - accuracy: 0.4584 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.4209 - accuracy: 0.5625 38/1563 [..............................] - ETA: 2s - loss: 1.4527 - accuracy: 0.4819 76/1563 [>.............................] - ETA: 2s - loss: 1.4919 - accuracy: 0.4807 114/1563 [=>............................] - ETA: 1s - loss: 1.5002 - accuracy: 0.4734 152/1563 [=>............................] - ETA: 1s - loss: 1.4916 - accuracy: 0.4786 190/1563 [==>...........................] - ETA: 1s - loss: 1.4984 - accuracy: 0.4752 228/1563 [===>..........................] - ETA: 1s - loss: 1.5009 - accuracy: 0.4749 266/1563 [====>.........................] - ETA: 1s - loss: 1.5057 - accuracy: 0.4726 304/1563 [====>.........................] - ETA: 1s - loss: 1.5025 - accuracy: 0.4746 342/1563 [=====>........................] - ETA: 1s - loss: 1.4987 - accuracy: 0.4768 380/1563 [======>.......................] - ETA: 1s - loss: 1.4986 - accuracy: 0.4764 418/1563 [=======>......................] - ETA: 1s - loss: 1.4976 - accuracy: 0.4757 456/1563 [=======>......................] - ETA: 1s - loss: 1.4973 - accuracy: 0.4743 494/1563 [========>.....................] - ETA: 1s - loss: 1.4951 - accuracy: 0.4746 532/1563 [=========>....................] - ETA: 1s - loss: 1.4958 - accuracy: 0.4740 570/1563 [=========>....................] - ETA: 1s - loss: 1.4934 - accuracy: 0.4738 608/1563 [==========>...................] - ETA: 1s - loss: 1.4916 - accuracy: 0.4737 645/1563 [===========>..................] - ETA: 1s - loss: 1.4911 - accuracy: 0.4734 683/1563 [============>.................] - ETA: 1s - loss: 1.4922 - accuracy: 0.4731 721/1563 [============>.................] - ETA: 1s - loss: 1.4907 - accuracy: 0.4732 759/1563 [=============>................] - ETA: 1s - loss: 1.4900 - accuracy: 0.4733 797/1563 [==============>...............] - ETA: 1s - loss: 1.4892 - accuracy: 0.4747 835/1563 [===============>..............] - ETA: 0s - loss: 1.4872 - accuracy: 0.4756 873/1563 [===============>..............] - ETA: 0s - loss: 1.4878 - accuracy: 0.4759 911/1563 [================>.............] - ETA: 0s - loss: 1.4881 - accuracy: 0.4763 949/1563 [=================>............] - ETA: 0s - loss: 1.4866 - accuracy: 0.4767 987/1563 [=================>............] - ETA: 0s - loss: 1.4860 - accuracy: 0.4772 1025/1563 [==================>...........] - ETA: 0s - loss: 1.4857 - accuracy: 0.4772 1063/1563 [===================>..........] - ETA: 0s - loss: 1.4842 - accuracy: 0.4773 1101/1563 [====================>.........] - ETA: 0s - loss: 1.4833 - accuracy: 0.4774 1139/1563 [====================>.........] - ETA: 0s - loss: 1.4836 - accuracy: 0.4776 1177/1563 [=====================>........] - ETA: 0s - loss: 1.4838 - accuracy: 0.4780 1214/1563 [======================>.......] - ETA: 0s - loss: 1.4835 - accuracy: 0.4782 1251/1563 [=======================>......] - ETA: 0s - loss: 1.4854 - accuracy: 0.4776 1289/1563 [=======================>......] - ETA: 0s - loss: 1.4831 - accuracy: 0.4781 1326/1563 [========================>.....] - ETA: 0s - loss: 1.4822 - accuracy: 0.4786 1364/1563 [=========================>....] - ETA: 0s - loss: 1.4816 - accuracy: 0.4784 1402/1563 [=========================>....] - ETA: 0s - loss: 1.4802 - accuracy: 0.4788 1440/1563 [==========================>...] - ETA: 0s - loss: 1.4792 - accuracy: 0.4793 1477/1563 [===========================>..] - ETA: 0s - loss: 1.4797 - accuracy: 0.4791 1514/1563 [============================>.] - ETA: 0s - loss: 1.4789 - accuracy: 0.4794 1552/1563 [============================>.] - ETA: 0s - loss: 1.4790 - accuracy: 0.4795 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4792 - accuracy: 0.4794 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.5872 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.4042 - accuracy: 0.4928 77/1563 [>.............................] - ETA: 2s - loss: 1.4165 - accuracy: 0.5024 115/1563 [=>............................] - ETA: 1s - loss: 1.4103 - accuracy: 0.5035 153/1563 [=>............................] - ETA: 1s - loss: 1.4198 - accuracy: 0.5039 191/1563 [==>...........................] - ETA: 1s - loss: 1.4250 - accuracy: 0.5016 228/1563 [===>..........................] - ETA: 1s - loss: 1.4237 - accuracy: 0.5014 265/1563 [====>.........................] - ETA: 1s - loss: 1.4284 - accuracy: 0.4998 303/1563 [====>.........................] - ETA: 1s - loss: 1.4342 - accuracy: 0.4974 340/1563 [=====>........................] - ETA: 1s - loss: 1.4365 - accuracy: 0.4974 378/1563 [======>.......................] - ETA: 1s - loss: 1.4349 - accuracy: 0.4983 416/1563 [======>.......................] - ETA: 1s - loss: 1.4331 - accuracy: 0.4969 454/1563 [=======>......................] - ETA: 1s - loss: 1.4337 - accuracy: 0.4974 492/1563 [========>.....................] - ETA: 1s - loss: 1.4348 - accuracy: 0.4952 530/1563 [=========>....................] - ETA: 1s - loss: 1.4348 - accuracy: 0.4962 568/1563 [=========>....................] - ETA: 1s - loss: 1.4328 - accuracy: 0.4966 606/1563 [==========>...................] - ETA: 1s - loss: 1.4325 - accuracy: 0.4967 644/1563 [===========>..................] - ETA: 1s - loss: 1.4319 - accuracy: 0.4968 682/1563 [============>.................] - ETA: 1s - loss: 1.4300 - accuracy: 0.4974 720/1563 [============>.................] - ETA: 1s - loss: 1.4300 - accuracy: 0.4968 758/1563 [=============>................] - ETA: 1s - loss: 1.4309 - accuracy: 0.4962 796/1563 [==============>...............] - ETA: 1s - loss: 1.4312 - accuracy: 0.4963 834/1563 [===============>..............] - ETA: 0s - loss: 1.4322 - accuracy: 0.4957 872/1563 [===============>..............] - ETA: 0s - loss: 1.4320 - accuracy: 0.4953 910/1563 [================>.............] - ETA: 0s - loss: 1.4321 - accuracy: 0.4949 947/1563 [=================>............] - ETA: 0s - loss: 1.4304 - accuracy: 0.4964 984/1563 [=================>............] - ETA: 0s - loss: 1.4302 - accuracy: 0.4961 1021/1563 [==================>...........] - ETA: 0s - loss: 1.4313 - accuracy: 0.4954 1058/1563 [===================>..........] - ETA: 0s - loss: 1.4322 - accuracy: 0.4953 1096/1563 [====================>.........] - ETA: 0s - loss: 1.4324 - accuracy: 0.4945 1134/1563 [====================>.........] - ETA: 0s - loss: 1.4311 - accuracy: 0.4949 1172/1563 [=====================>........] - ETA: 0s - loss: 1.4327 - accuracy: 0.4939 1210/1563 [======================>.......] - ETA: 0s - loss: 1.4341 - accuracy: 0.4934 1247/1563 [======================>.......] - ETA: 0s - loss: 1.4348 - accuracy: 0.4929 1285/1563 [=======================>......] - ETA: 0s - loss: 1.4349 - accuracy: 0.4928 1323/1563 [========================>.....] - ETA: 0s - loss: 1.4355 - accuracy: 0.4930 1360/1563 [=========================>....] - ETA: 0s - loss: 1.4347 - accuracy: 0.4931 1398/1563 [=========================>....] - ETA: 0s - loss: 1.4337 - accuracy: 0.4938 1436/1563 [==========================>...] - ETA: 0s - loss: 1.4337 - accuracy: 0.4936 1474/1563 [===========================>..] - ETA: 0s - loss: 1.4325 - accuracy: 0.4944 1512/1563 [============================>.] - ETA: 0s - loss: 1.4330 - accuracy: 0.4942 1550/1563 [============================>.] - ETA: 0s - loss: 1.4320 - accuracy: 0.4947 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4315 - accuracy: 0.4949 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.3437 - accuracy: 0.5000 38/1563 [..............................] - ETA: 2s - loss: 1.3705 - accuracy: 0.5181 75/1563 [>.............................] - ETA: 2s - loss: 1.3786 - accuracy: 0.5146 113/1563 [=>............................] - ETA: 1s - loss: 1.3808 - accuracy: 0.5122 151/1563 [=>............................] - ETA: 1s - loss: 1.3895 - accuracy: 0.5072 189/1563 [==>...........................] - ETA: 1s - loss: 1.3919 - accuracy: 0.5081 227/1563 [===>..........................] - ETA: 1s - loss: 1.3957 - accuracy: 0.5105 265/1563 [====>.........................] - ETA: 1s - loss: 1.3975 - accuracy: 0.5098 303/1563 [====>.........................] - ETA: 1s - loss: 1.3980 - accuracy: 0.5102 341/1563 [=====>........................] - ETA: 1s - loss: 1.3941 - accuracy: 0.5126 379/1563 [======>.......................] - ETA: 1s - loss: 1.3937 - accuracy: 0.5123 417/1563 [=======>......................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5105 455/1563 [=======>......................] - ETA: 1s - loss: 1.3955 - accuracy: 0.5104 493/1563 [========>.....................] - ETA: 1s - loss: 1.3926 - accuracy: 0.5115 531/1563 [=========>....................] - ETA: 1s - loss: 1.3932 - accuracy: 0.5104 569/1563 [=========>....................] - ETA: 1s - loss: 1.3929 - accuracy: 0.5103 607/1563 [==========>...................] - ETA: 1s - loss: 1.3914 - accuracy: 0.5099 645/1563 [===========>..................] - ETA: 1s - loss: 1.3892 - accuracy: 0.5113 683/1563 [============>.................] - ETA: 1s - loss: 1.3923 - accuracy: 0.5092 721/1563 [============>.................] - ETA: 1s - loss: 1.3906 - accuracy: 0.5097 759/1563 [=============>................] - ETA: 1s - loss: 1.3893 - accuracy: 0.5105 797/1563 [==============>...............] - ETA: 1s - loss: 1.3902 - accuracy: 0.5109 835/1563 [===============>..............] - ETA: 0s - loss: 1.3877 - accuracy: 0.5112 873/1563 [===============>..............] - ETA: 0s - loss: 1.3873 - accuracy: 0.5118 910/1563 [================>.............] - ETA: 0s - loss: 1.3851 - accuracy: 0.5123 948/1563 [=================>............] - ETA: 0s - loss: 1.3875 - accuracy: 0.5116 985/1563 [=================>............] - ETA: 0s - loss: 1.3874 - accuracy: 0.5115 1023/1563 [==================>...........] - ETA: 0s - loss: 1.3879 - accuracy: 0.5113 1061/1563 [===================>..........] - ETA: 0s - loss: 1.3881 - accuracy: 0.5115 1099/1563 [====================>.........] - ETA: 0s - loss: 1.3886 - accuracy: 0.5116 1137/1563 [====================>.........] - ETA: 0s - loss: 1.3902 - accuracy: 0.5115 1175/1563 [=====================>........] - ETA: 0s - loss: 1.3888 - accuracy: 0.5120 1213/1563 [======================>.......] - ETA: 0s - loss: 1.3896 - accuracy: 0.5112 1251/1563 [=======================>......] - ETA: 0s - loss: 1.3890 - accuracy: 0.5115 1289/1563 [=======================>......] - ETA: 0s - loss: 1.3913 - accuracy: 0.5109 1327/1563 [========================>.....] - ETA: 0s - loss: 1.3927 - accuracy: 0.5107 1365/1563 [=========================>....] - ETA: 0s - loss: 1.3907 - accuracy: 0.5113 1403/1563 [=========================>....] - ETA: 0s - loss: 1.3910 - accuracy: 0.5111 1441/1563 [==========================>...] - ETA: 0s - loss: 1.3908 - accuracy: 0.5116 1479/1563 [===========================>..] - ETA: 0s - loss: 1.3900 - accuracy: 0.5122 1516/1563 [============================>.] - ETA: 0s - loss: 1.3893 - accuracy: 0.5128 1553/1563 [============================>.] - ETA: 0s - loss: 1.3897 - accuracy: 0.5131 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3895 - accuracy: 0.5132 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.1149 - accuracy: 0.6250 39/1563 [..............................] - ETA: 2s - loss: 1.3790 - accuracy: 0.4968 76/1563 [>.............................] - ETA: 2s - loss: 1.3333 - accuracy: 0.5189 114/1563 [=>............................] - ETA: 1s - loss: 1.3295 - accuracy: 0.5269 152/1563 [=>............................] - ETA: 1s - loss: 1.3448 - accuracy: 0.5226 190/1563 [==>...........................] - ETA: 1s - loss: 1.3459 - accuracy: 0.5245 228/1563 [===>..........................] - ETA: 1s - loss: 1.3473 - accuracy: 0.5260 265/1563 [====>.........................] - ETA: 1s - loss: 1.3436 - accuracy: 0.5289 302/1563 [====>.........................] - ETA: 1s - loss: 1.3406 - accuracy: 0.5293 339/1563 [=====>........................] - ETA: 1s - loss: 1.3441 - accuracy: 0.5286 376/1563 [======>.......................] - ETA: 1s - loss: 1.3482 - accuracy: 0.5277 414/1563 [======>.......................] - ETA: 1s - loss: 1.3536 - accuracy: 0.5265 452/1563 [=======>......................] - ETA: 1s - loss: 1.3554 - accuracy: 0.5245 490/1563 [========>.....................] - ETA: 1s - loss: 1.3560 - accuracy: 0.5246 528/1563 [=========>....................] - ETA: 1s - loss: 1.3504 - accuracy: 0.5263 566/1563 [=========>....................] - ETA: 1s - loss: 1.3537 - accuracy: 0.5253 604/1563 [==========>...................] - ETA: 1s - loss: 1.3569 - accuracy: 0.5226 642/1563 [===========>..................] - ETA: 1s - loss: 1.3581 - accuracy: 0.5220 679/1563 [============>.................] - ETA: 1s - loss: 1.3571 - accuracy: 0.5219 717/1563 [============>.................] - ETA: 1s - loss: 1.3533 - accuracy: 0.5226 755/1563 [=============>................] - ETA: 1s - loss: 1.3552 - accuracy: 0.5227 793/1563 [==============>...............] - ETA: 1s - loss: 1.3563 - accuracy: 0.5222 831/1563 [==============>...............] - ETA: 0s - loss: 1.3568 - accuracy: 0.5226 869/1563 [===============>..............] - ETA: 0s - loss: 1.3590 - accuracy: 0.5214 907/1563 [================>.............] - ETA: 0s - loss: 1.3576 - accuracy: 0.5222 944/1563 [=================>............] - ETA: 0s - loss: 1.3577 - accuracy: 0.5214 982/1563 [=================>............] - ETA: 0s - loss: 1.3574 - accuracy: 0.5216 1020/1563 [==================>...........] - ETA: 0s - loss: 1.3593 - accuracy: 0.5216 1058/1563 [===================>..........] - ETA: 0s - loss: 1.3585 - accuracy: 0.5219 1096/1563 [====================>.........] - ETA: 0s - loss: 1.3584 - accuracy: 0.5220 1134/1563 [====================>.........] - ETA: 0s - loss: 1.3577 - accuracy: 0.5222 1171/1563 [=====================>........] - ETA: 0s - loss: 1.3565 - accuracy: 0.5226 1209/1563 [======================>.......] - ETA: 0s - loss: 1.3562 - accuracy: 0.5229 1246/1563 [======================>.......] - ETA: 0s - loss: 1.3555 - accuracy: 0.5230 1284/1563 [=======================>......] - ETA: 0s - loss: 1.3547 - accuracy: 0.5233 1322/1563 [========================>.....] - ETA: 0s - loss: 1.3536 - accuracy: 0.5235 1360/1563 [=========================>....] - ETA: 0s - loss: 1.3531 - accuracy: 0.5238 1399/1563 [=========================>....] - ETA: 0s - loss: 1.3528 - accuracy: 0.5238 1437/1563 [==========================>...] - ETA: 0s - loss: 1.3523 - accuracy: 0.5239 1475/1563 [===========================>..] - ETA: 0s - loss: 1.3527 - accuracy: 0.5239 1513/1563 [============================>.] - ETA: 0s - loss: 1.3524 - accuracy: 0.5240 1551/1563 [============================>.] - ETA: 0s - loss: 1.3525 - accuracy: 0.5240 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3524 - accuracy: 0.5238 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.2660 - accuracy: 0.5938 39/1563 [..............................] - ETA: 2s - loss: 1.3038 - accuracy: 0.5401 77/1563 [>.............................] - ETA: 1s - loss: 1.2802 - accuracy: 0.5414 115/1563 [=>............................] - ETA: 1s - loss: 1.2991 - accuracy: 0.5321 153/1563 [=>............................] - ETA: 1s - loss: 1.3168 - accuracy: 0.5323 191/1563 [==>...........................] - ETA: 1s - loss: 1.3136 - accuracy: 0.5366 228/1563 [===>..........................] - ETA: 1s - loss: 1.3172 - accuracy: 0.5328 266/1563 [====>.........................] - ETA: 1s - loss: 1.3198 - accuracy: 0.5335 304/1563 [====>.........................] - ETA: 1s - loss: 1.3175 - accuracy: 0.5356 340/1563 [=====>........................] - ETA: 1s - loss: 1.3230 - accuracy: 0.5344 378/1563 [======>.......................] - ETA: 1s - loss: 1.3194 - accuracy: 0.5335 416/1563 [======>.......................] - ETA: 1s - loss: 1.3173 - accuracy: 0.5340 454/1563 [=======>......................] - ETA: 1s - loss: 1.3212 - accuracy: 0.5312 492/1563 [========>.....................] - ETA: 1s - loss: 1.3216 - accuracy: 0.5316 530/1563 [=========>....................] - ETA: 1s - loss: 1.3204 - accuracy: 0.5320 568/1563 [=========>....................] - ETA: 1s - loss: 1.3202 - accuracy: 0.5331 605/1563 [==========>...................] - ETA: 1s - loss: 1.3190 - accuracy: 0.5328 643/1563 [===========>..................] - ETA: 1s - loss: 1.3174 - accuracy: 0.5338 681/1563 [============>.................] - ETA: 1s - loss: 1.3186 - accuracy: 0.5340 719/1563 [============>.................] - ETA: 1s - loss: 1.3197 - accuracy: 0.5339 757/1563 [=============>................] - ETA: 1s - loss: 1.3201 - accuracy: 0.5338 795/1563 [==============>...............] - ETA: 1s - loss: 1.3186 - accuracy: 0.5351 833/1563 [==============>...............] - ETA: 0s - loss: 1.3170 - accuracy: 0.5354 871/1563 [===============>..............] - ETA: 0s - loss: 1.3186 - accuracy: 0.5355 909/1563 [================>.............] - ETA: 0s - loss: 1.3192 - accuracy: 0.5361 947/1563 [=================>............] - ETA: 0s - loss: 1.3186 - accuracy: 0.5372 985/1563 [=================>............] - ETA: 0s - loss: 1.3199 - accuracy: 0.5366 1023/1563 [==================>...........] - ETA: 0s - loss: 1.3191 - accuracy: 0.5371 1061/1563 [===================>..........] - ETA: 0s - loss: 1.3182 - accuracy: 0.5370 1099/1563 [====================>.........] - ETA: 0s - loss: 1.3192 - accuracy: 0.5363 1137/1563 [====================>.........] - ETA: 0s - loss: 1.3176 - accuracy: 0.5367 1175/1563 [=====================>........] - ETA: 0s - loss: 1.3169 - accuracy: 0.5372 1213/1563 [======================>.......] - ETA: 0s - loss: 1.3171 - accuracy: 0.5372 1250/1563 [======================>.......] - ETA: 0s - loss: 1.3162 - accuracy: 0.5376 1288/1563 [=======================>......] - ETA: 0s - loss: 1.3156 - accuracy: 0.5379 1326/1563 [========================>.....] - ETA: 0s - loss: 1.3163 - accuracy: 0.5375 1364/1563 [=========================>....] - ETA: 0s - loss: 1.3152 - accuracy: 0.5381 1402/1563 [=========================>....] - ETA: 0s - loss: 1.3141 - accuracy: 0.5381 1440/1563 [==========================>...] - ETA: 0s - loss: 1.3162 - accuracy: 0.5376 1478/1563 [===========================>..] - ETA: 0s - loss: 1.3155 - accuracy: 0.5373 1516/1563 [============================>.] - ETA: 0s - loss: 1.3157 - accuracy: 0.5370 1554/1563 [============================>.] - ETA: 0s - loss: 1.3161 - accuracy: 0.5367 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3165 - accuracy: 0.5365 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.6358 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.3514 - accuracy: 0.5192 77/1563 [>.............................] - ETA: 1s - loss: 1.3139 - accuracy: 0.5414 115/1563 [=>............................] - ETA: 1s - loss: 1.3075 - accuracy: 0.5446 153/1563 [=>............................] - ETA: 1s - loss: 1.3046 - accuracy: 0.5468 191/1563 [==>...........................] - ETA: 1s - loss: 1.3061 - accuracy: 0.5448 229/1563 [===>..........................] - ETA: 1s - loss: 1.3023 - accuracy: 0.5461 267/1563 [====>.........................] - ETA: 1s - loss: 1.3049 - accuracy: 0.5440 305/1563 [====>.........................] - ETA: 1s - loss: 1.3022 - accuracy: 0.5443 343/1563 [=====>........................] - ETA: 1s - loss: 1.2986 - accuracy: 0.5452 381/1563 [======>.......................] - ETA: 1s - loss: 1.3013 - accuracy: 0.5431 419/1563 [=======>......................] - ETA: 1s - loss: 1.2989 - accuracy: 0.5450 457/1563 [=======>......................] - ETA: 1s - loss: 1.2994 - accuracy: 0.5452 495/1563 [========>.....................] - ETA: 1s - loss: 1.2987 - accuracy: 0.5446 532/1563 [=========>....................] - ETA: 1s - loss: 1.3000 - accuracy: 0.5439 570/1563 [=========>....................] - ETA: 1s - loss: 1.2973 - accuracy: 0.5440 608/1563 [==========>...................] - ETA: 1s - loss: 1.2949 - accuracy: 0.5448 646/1563 [===========>..................] - ETA: 1s - loss: 1.2954 - accuracy: 0.5451 684/1563 [============>.................] - ETA: 1s - loss: 1.2938 - accuracy: 0.5455 722/1563 [============>.................] - ETA: 1s - loss: 1.2923 - accuracy: 0.5462 760/1563 [=============>................] - ETA: 1s - loss: 1.2930 - accuracy: 0.5463 798/1563 [==============>...............] - ETA: 1s - loss: 1.2942 - accuracy: 0.5461 835/1563 [===============>..............] - ETA: 0s - loss: 1.2913 - accuracy: 0.5472 872/1563 [===============>..............] - ETA: 0s - loss: 1.2916 - accuracy: 0.5468 910/1563 [================>.............] - ETA: 0s - loss: 1.2902 - accuracy: 0.5474 948/1563 [=================>............] - ETA: 0s - loss: 1.2902 - accuracy: 0.5474 986/1563 [=================>............] - ETA: 0s - loss: 1.2891 - accuracy: 0.5471 1024/1563 [==================>...........] - ETA: 0s - loss: 1.2868 - accuracy: 0.5481 1062/1563 [===================>..........] - ETA: 0s - loss: 1.2861 - accuracy: 0.5489 1100/1563 [====================>.........] - ETA: 0s - loss: 1.2861 - accuracy: 0.5490 1138/1563 [====================>.........] - ETA: 0s - loss: 1.2857 - accuracy: 0.5490 1175/1563 [=====================>........] - ETA: 0s - loss: 1.2859 - accuracy: 0.5485 1213/1563 [======================>.......] - ETA: 0s - loss: 1.2856 - accuracy: 0.5484 1251/1563 [=======================>......] - ETA: 0s - loss: 1.2846 - accuracy: 0.5487 1289/1563 [=======================>......] - ETA: 0s - loss: 1.2862 - accuracy: 0.5486 1327/1563 [========================>.....] - ETA: 0s - loss: 1.2856 - accuracy: 0.5491 1365/1563 [=========================>....] - ETA: 0s - loss: 1.2852 - accuracy: 0.5488 1403/1563 [=========================>....] - ETA: 0s - loss: 1.2862 - accuracy: 0.5483 1441/1563 [==========================>...] - ETA: 0s - loss: 1.2850 - accuracy: 0.5488 1479/1563 [===========================>..] - ETA: 0s - loss: 1.2845 - accuracy: 0.5489 1517/1563 [============================>.] - ETA: 0s - loss: 1.2837 - accuracy: 0.5487 1555/1563 [============================>.] - ETA: 0s - loss: 1.2841 - accuracy: 0.5483 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2840 - accuracy: 0.5483 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.2325 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.2643 - accuracy: 0.5593 77/1563 [>.............................] - ETA: 1s - loss: 1.2671 - accuracy: 0.5556 115/1563 [=>............................] - ETA: 1s - loss: 1.2489 - accuracy: 0.5554 153/1563 [=>............................] - ETA: 1s - loss: 1.2436 - accuracy: 0.5580 191/1563 [==>...........................] - ETA: 1s - loss: 1.2600 - accuracy: 0.5525 229/1563 [===>..........................] - ETA: 1s - loss: 1.2562 - accuracy: 0.5535 267/1563 [====>.........................] - ETA: 1s - loss: 1.2552 - accuracy: 0.5563 304/1563 [====>.........................] - ETA: 1s - loss: 1.2624 - accuracy: 0.5552 341/1563 [=====>........................] - ETA: 1s - loss: 1.2635 - accuracy: 0.5540 379/1563 [======>.......................] - ETA: 1s - loss: 1.2586 - accuracy: 0.5557 416/1563 [======>.......................] - ETA: 1s - loss: 1.2628 - accuracy: 0.5540 454/1563 [=======>......................] - ETA: 1s - loss: 1.2661 - accuracy: 0.5541 492/1563 [========>.....................] - ETA: 1s - loss: 1.2654 - accuracy: 0.5561 530/1563 [=========>....................] - ETA: 1s - loss: 1.2671 - accuracy: 0.5547 568/1563 [=========>....................] - ETA: 1s - loss: 1.2627 - accuracy: 0.5553 606/1563 [==========>...................] - ETA: 1s - loss: 1.2628 - accuracy: 0.5551 643/1563 [===========>..................] - ETA: 1s - loss: 1.2646 - accuracy: 0.5554 681/1563 [============>.................] - ETA: 1s - loss: 1.2623 - accuracy: 0.5556 719/1563 [============>.................] - ETA: 1s - loss: 1.2614 - accuracy: 0.5546 757/1563 [=============>................] - ETA: 1s - loss: 1.2606 - accuracy: 0.5553 795/1563 [==============>...............] - ETA: 1s - loss: 1.2602 - accuracy: 0.5553 833/1563 [==============>...............] - ETA: 0s - loss: 1.2611 - accuracy: 0.5544 871/1563 [===============>..............] - ETA: 0s - loss: 1.2614 - accuracy: 0.5543 909/1563 [================>.............] - ETA: 0s - loss: 1.2622 - accuracy: 0.5545 947/1563 [=================>............] - ETA: 0s - loss: 1.2628 - accuracy: 0.5551 985/1563 [=================>............] - ETA: 0s - loss: 1.2618 - accuracy: 0.5563 1023/1563 [==================>...........] - ETA: 0s - loss: 1.2602 - accuracy: 0.5572 1060/1563 [===================>..........] - ETA: 0s - loss: 1.2599 - accuracy: 0.5574 1098/1563 [====================>.........] - ETA: 0s - loss: 1.2588 - accuracy: 0.5582 1136/1563 [====================>.........] - ETA: 0s - loss: 1.2571 - accuracy: 0.5588 1174/1563 [=====================>........] - ETA: 0s - loss: 1.2557 - accuracy: 0.5590 1211/1563 [======================>.......] - ETA: 0s - loss: 1.2554 - accuracy: 0.5588 1249/1563 [======================>.......] - ETA: 0s - loss: 1.2544 - accuracy: 0.5594 1287/1563 [=======================>......] - ETA: 0s - loss: 1.2548 - accuracy: 0.5593 1325/1563 [========================>.....] - ETA: 0s - loss: 1.2544 - accuracy: 0.5591 1363/1563 [=========================>....] - ETA: 0s - loss: 1.2559 - accuracy: 0.5585 1401/1563 [=========================>....] - ETA: 0s - loss: 1.2561 - accuracy: 0.5585 1439/1563 [==========================>...] - ETA: 0s - loss: 1.2557 - accuracy: 0.5582 1477/1563 [===========================>..] - ETA: 0s - loss: 1.2555 - accuracy: 0.5582 1515/1563 [============================>.] - ETA: 0s - loss: 1.2557 - accuracy: 0.5580 1553/1563 [============================>.] - ETA: 0s - loss: 1.2538 - accuracy: 0.5586 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2531 - accuracy: 0.5588 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 37.25028657913208 hydro05 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins4177595130164554911.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS