Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 95795 queued and waiting for resources srun: job 95795 has been allocated resources 2024-10-28 07:43:07.826208: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-28 07:43:32.191531: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-10-28 07:44:06.757844: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-10-28 07:44:06.758803: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-10-28 07:44:06.758811: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-10-28 07:45:01.149549: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-28 07:45:03.791532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-10-28 07:45:11.006664: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:40:05 - loss: 2.3634 - accuracy: 0.0938 37/1563 [..............................] - ETA: 2s - loss: 2.2428 - accuracy: 0.1875  74/1563 [>.............................] - ETA: 2s - loss: 2.1808 - accuracy: 0.2048 112/1563 [=>............................] - ETA: 1s - loss: 2.1297 - accuracy: 0.2252 150/1563 [=>............................] - ETA: 1s - loss: 2.0961 - accuracy: 0.2385 188/1563 [==>...........................] - ETA: 1s - loss: 2.0677 - accuracy: 0.2510 225/1563 [===>..........................] - ETA: 1s - loss: 2.0360 - accuracy: 0.2631 263/1563 [====>.........................] - ETA: 1s - loss: 2.0248 - accuracy: 0.2670 301/1563 [====>.........................] - ETA: 1s - loss: 2.0062 - accuracy: 0.2770 339/1563 [=====>........................] - ETA: 1s - loss: 1.9933 - accuracy: 0.2818 377/1563 [======>.......................] - ETA: 1s - loss: 1.9852 - accuracy: 0.2864 415/1563 [======>.......................] - ETA: 1s - loss: 1.9708 - accuracy: 0.2906 453/1563 [=======>......................] - ETA: 1s - loss: 1.9641 - accuracy: 0.2935 491/1563 [========>.....................] - ETA: 1s - loss: 1.9551 - accuracy: 0.2956 529/1563 [=========>....................] - ETA: 1s - loss: 1.9478 - accuracy: 0.2976 567/1563 [=========>....................] - ETA: 1s - loss: 1.9416 - accuracy: 0.3007 605/1563 [==========>...................] - ETA: 1s - loss: 1.9305 - accuracy: 0.3053 643/1563 [===========>..................] - ETA: 1s - loss: 1.9208 - accuracy: 0.3088 681/1563 [============>.................] - ETA: 1s - loss: 1.9129 - accuracy: 0.3118 719/1563 [============>.................] - ETA: 1s - loss: 1.9058 - accuracy: 0.3151 757/1563 [=============>................] - ETA: 1s - loss: 1.9012 - accuracy: 0.3170 795/1563 [==============>...............] - ETA: 1s - loss: 1.8950 - accuracy: 0.3199 833/1563 [==============>...............] - ETA: 0s - loss: 1.8899 - accuracy: 0.3216 871/1563 [===============>..............] - ETA: 0s - loss: 1.8844 - accuracy: 0.3248 909/1563 [================>.............] - ETA: 0s - loss: 1.8782 - accuracy: 0.3277 947/1563 [=================>............] - ETA: 0s - loss: 1.8731 - accuracy: 0.3294 985/1563 [=================>............] - ETA: 0s - loss: 1.8683 - accuracy: 0.3319 1023/1563 [==================>...........] - ETA: 0s - loss: 1.8638 - accuracy: 0.3334 1061/1563 [===================>..........] - ETA: 0s - loss: 1.8583 - accuracy: 0.3355 1099/1563 [====================>.........] - ETA: 0s - loss: 1.8536 - accuracy: 0.3380 1137/1563 [====================>.........] - ETA: 0s - loss: 1.8495 - accuracy: 0.3397 1175/1563 [=====================>........] - ETA: 0s - loss: 1.8459 - accuracy: 0.3409 1213/1563 [======================>.......] - ETA: 0s - loss: 1.8426 - accuracy: 0.3427 1251/1563 [=======================>......] - ETA: 0s - loss: 1.8402 - accuracy: 0.3439 1289/1563 [=======================>......] - ETA: 0s - loss: 1.8362 - accuracy: 0.3459 1327/1563 [========================>.....] - ETA: 0s - loss: 1.8326 - accuracy: 0.3469 1365/1563 [=========================>....] - ETA: 0s - loss: 1.8300 - accuracy: 0.3479 1403/1563 [=========================>....] - ETA: 0s - loss: 1.8272 - accuracy: 0.3490 1441/1563 [==========================>...] - ETA: 0s - loss: 1.8242 - accuracy: 0.3504 1479/1563 [===========================>..] - ETA: 0s - loss: 1.8212 - accuracy: 0.3519 1517/1563 [============================>.] - ETA: 0s - loss: 1.8167 - accuracy: 0.3536 1555/1563 [============================>.] - ETA: 0s - loss: 1.8128 - accuracy: 0.3550 1563/1563 [==============================] - 6s 1ms/step - loss: 1.8124 - accuracy: 0.3551 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.9957 - accuracy: 0.3125 39/1563 [..............................] - ETA: 2s - loss: 1.6620 - accuracy: 0.4159 77/1563 [>.............................] - ETA: 1s - loss: 1.6504 - accuracy: 0.4156 115/1563 [=>............................] - ETA: 1s - loss: 1.6642 - accuracy: 0.4168 153/1563 [=>............................] - ETA: 1s - loss: 1.6619 - accuracy: 0.4140 191/1563 [==>...........................] - ETA: 1s - loss: 1.6559 - accuracy: 0.4157 229/1563 [===>..........................] - ETA: 1s - loss: 1.6586 - accuracy: 0.4139 267/1563 [====>.........................] - ETA: 1s - loss: 1.6543 - accuracy: 0.4190 305/1563 [====>.........................] - ETA: 1s - loss: 1.6532 - accuracy: 0.4187 343/1563 [=====>........................] - ETA: 1s - loss: 1.6484 - accuracy: 0.4208 381/1563 [======>.......................] - ETA: 1s - loss: 1.6481 - accuracy: 0.4210 419/1563 [=======>......................] - ETA: 1s - loss: 1.6421 - accuracy: 0.4224 457/1563 [=======>......................] - ETA: 1s - loss: 1.6427 - accuracy: 0.4220 495/1563 [========>.....................] - ETA: 1s - loss: 1.6449 - accuracy: 0.4227 533/1563 [=========>....................] - ETA: 1s - loss: 1.6430 - accuracy: 0.4227 571/1563 [=========>....................] - ETA: 1s - loss: 1.6440 - accuracy: 0.4217 609/1563 [==========>...................] - ETA: 1s - loss: 1.6433 - accuracy: 0.4215 647/1563 [===========>..................] - ETA: 1s - loss: 1.6425 - accuracy: 0.4226 685/1563 [============>.................] - ETA: 1s - loss: 1.6425 - accuracy: 0.4225 723/1563 [============>.................] - ETA: 1s - loss: 1.6417 - accuracy: 0.4222 761/1563 [=============>................] - ETA: 1s - loss: 1.6396 - accuracy: 0.4230 799/1563 [==============>...............] - ETA: 1s - loss: 1.6369 - accuracy: 0.4234 837/1563 [===============>..............] - ETA: 0s - loss: 1.6370 - accuracy: 0.4233 875/1563 [===============>..............] - ETA: 0s - loss: 1.6365 - accuracy: 0.4234 913/1563 [================>.............] - ETA: 0s - loss: 1.6371 - accuracy: 0.4231 951/1563 [=================>............] - ETA: 0s - loss: 1.6354 - accuracy: 0.4239 989/1563 [=================>............] - ETA: 0s - loss: 1.6347 - accuracy: 0.4238 1027/1563 [==================>...........] - ETA: 0s - loss: 1.6340 - accuracy: 0.4237 1065/1563 [===================>..........] - ETA: 0s - loss: 1.6342 - accuracy: 0.4235 1103/1563 [====================>.........] - ETA: 0s - loss: 1.6327 - accuracy: 0.4237 1141/1563 [====================>.........] - ETA: 0s - loss: 1.6315 - accuracy: 0.4236 1179/1563 [=====================>........] - ETA: 0s - loss: 1.6301 - accuracy: 0.4244 1217/1563 [======================>.......] - ETA: 0s - loss: 1.6294 - accuracy: 0.4244 1255/1563 [=======================>......] - ETA: 0s - loss: 1.6292 - accuracy: 0.4246 1293/1563 [=======================>......] - ETA: 0s - loss: 1.6288 - accuracy: 0.4247 1331/1563 [========================>.....] - ETA: 0s - loss: 1.6284 - accuracy: 0.4250 1369/1563 [=========================>....] - ETA: 0s - loss: 1.6279 - accuracy: 0.4257 1407/1563 [==========================>...] - ETA: 0s - loss: 1.6273 - accuracy: 0.4259 1445/1563 [==========================>...] - ETA: 0s - loss: 1.6270 - accuracy: 0.4252 1483/1563 [===========================>..] - ETA: 0s - loss: 1.6245 - accuracy: 0.4262 1521/1563 [============================>.] - ETA: 0s - loss: 1.6236 - accuracy: 0.4268 1559/1563 [============================>.] - ETA: 0s - loss: 1.6223 - accuracy: 0.4272 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6219 - accuracy: 0.4273 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.6275 - accuracy: 0.5000 39/1563 [..............................] - ETA: 2s - loss: 1.5296 - accuracy: 0.4728 77/1563 [>.............................] - ETA: 1s - loss: 1.5540 - accuracy: 0.4521 115/1563 [=>............................] - ETA: 1s - loss: 1.5523 - accuracy: 0.4484 153/1563 [=>............................] - ETA: 1s - loss: 1.5607 - accuracy: 0.4481 191/1563 [==>...........................] - ETA: 1s - loss: 1.5607 - accuracy: 0.4460 229/1563 [===>..........................] - ETA: 1s - loss: 1.5579 - accuracy: 0.4443 267/1563 [====>.........................] - ETA: 1s - loss: 1.5545 - accuracy: 0.4439 305/1563 [====>.........................] - ETA: 1s - loss: 1.5524 - accuracy: 0.4464 343/1563 [=====>........................] - ETA: 1s - loss: 1.5585 - accuracy: 0.4440 381/1563 [======>.......................] - ETA: 1s - loss: 1.5572 - accuracy: 0.4450 419/1563 [=======>......................] - ETA: 1s - loss: 1.5607 - accuracy: 0.4429 457/1563 [=======>......................] - ETA: 1s - loss: 1.5604 - accuracy: 0.4419 495/1563 [========>.....................] - ETA: 1s - loss: 1.5623 - accuracy: 0.4415 533/1563 [=========>....................] - ETA: 1s - loss: 1.5612 - accuracy: 0.4420 571/1563 [=========>....................] - ETA: 1s - loss: 1.5592 - accuracy: 0.4435 609/1563 [==========>...................] - ETA: 1s - loss: 1.5562 - accuracy: 0.4454 647/1563 [===========>..................] - ETA: 1s - loss: 1.5549 - accuracy: 0.4475 685/1563 [============>.................] - ETA: 1s - loss: 1.5562 - accuracy: 0.4465 723/1563 [============>.................] - ETA: 1s - loss: 1.5555 - accuracy: 0.4473 761/1563 [=============>................] - ETA: 1s - loss: 1.5525 - accuracy: 0.4486 799/1563 [==============>...............] - ETA: 1s - loss: 1.5525 - accuracy: 0.4490 837/1563 [===============>..............] - ETA: 0s - loss: 1.5511 - accuracy: 0.4501 875/1563 [===============>..............] - ETA: 0s - loss: 1.5513 - accuracy: 0.4505 913/1563 [================>.............] - ETA: 0s - loss: 1.5481 - accuracy: 0.4526 951/1563 [=================>............] - ETA: 0s - loss: 1.5480 - accuracy: 0.4530 989/1563 [=================>............] - ETA: 0s - loss: 1.5465 - accuracy: 0.4538 1027/1563 [==================>...........] - ETA: 0s - loss: 1.5475 - accuracy: 0.4536 1065/1563 [===================>..........] - ETA: 0s - loss: 1.5465 - accuracy: 0.4541 1103/1563 [====================>.........] - ETA: 0s - loss: 1.5477 - accuracy: 0.4540 1141/1563 [====================>.........] - ETA: 0s - loss: 1.5460 - accuracy: 0.4540 1179/1563 [=====================>........] - ETA: 0s - loss: 1.5453 - accuracy: 0.4547 1217/1563 [======================>.......] - ETA: 0s - loss: 1.5466 - accuracy: 0.4540 1255/1563 [=======================>......] - ETA: 0s - loss: 1.5454 - accuracy: 0.4540 1293/1563 [=======================>......] - ETA: 0s - loss: 1.5450 - accuracy: 0.4545 1331/1563 [========================>.....] - ETA: 0s - loss: 1.5436 - accuracy: 0.4555 1369/1563 [=========================>....] - ETA: 0s - loss: 1.5434 - accuracy: 0.4558 1407/1563 [==========================>...] - ETA: 0s - loss: 1.5442 - accuracy: 0.4557 1445/1563 [==========================>...] - ETA: 0s - loss: 1.5428 - accuracy: 0.4563 1483/1563 [===========================>..] - ETA: 0s - loss: 1.5432 - accuracy: 0.4562 1521/1563 [============================>.] - ETA: 0s - loss: 1.5425 - accuracy: 0.4567 1559/1563 [============================>.] - ETA: 0s - loss: 1.5415 - accuracy: 0.4571 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5417 - accuracy: 0.4572 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.1234 - accuracy: 0.6875 39/1563 [..............................] - ETA: 2s - loss: 1.4751 - accuracy: 0.4832 77/1563 [>.............................] - ETA: 1s - loss: 1.4719 - accuracy: 0.4850 115/1563 [=>............................] - ETA: 1s - loss: 1.4794 - accuracy: 0.4851 153/1563 [=>............................] - ETA: 1s - loss: 1.4807 - accuracy: 0.4816 191/1563 [==>...........................] - ETA: 1s - loss: 1.4819 - accuracy: 0.4833 229/1563 [===>..........................] - ETA: 1s - loss: 1.4748 - accuracy: 0.4844 267/1563 [====>.........................] - ETA: 1s - loss: 1.4735 - accuracy: 0.4843 305/1563 [====>.........................] - ETA: 1s - loss: 1.4788 - accuracy: 0.4823 343/1563 [=====>........................] - ETA: 1s - loss: 1.4776 - accuracy: 0.4842 381/1563 [======>.......................] - ETA: 1s - loss: 1.4769 - accuracy: 0.4853 419/1563 [=======>......................] - ETA: 1s - loss: 1.4784 - accuracy: 0.4849 457/1563 [=======>......................] - ETA: 1s - loss: 1.4818 - accuracy: 0.4843 495/1563 [========>.....................] - ETA: 1s - loss: 1.4803 - accuracy: 0.4847 533/1563 [=========>....................] - ETA: 1s - loss: 1.4843 - accuracy: 0.4823 571/1563 [=========>....................] - ETA: 1s - loss: 1.4845 - accuracy: 0.4813 609/1563 [==========>...................] - ETA: 1s - loss: 1.4847 - accuracy: 0.4805 647/1563 [===========>..................] - ETA: 1s - loss: 1.4892 - accuracy: 0.4790 685/1563 [============>.................] - ETA: 1s - loss: 1.4894 - accuracy: 0.4787 723/1563 [============>.................] - ETA: 1s - loss: 1.4916 - accuracy: 0.4774 761/1563 [=============>................] - ETA: 1s - loss: 1.4875 - accuracy: 0.4793 799/1563 [==============>...............] - ETA: 1s - loss: 1.4845 - accuracy: 0.4798 837/1563 [===============>..............] - ETA: 0s - loss: 1.4837 - accuracy: 0.4795 875/1563 [===============>..............] - ETA: 0s - loss: 1.4825 - accuracy: 0.4805 913/1563 [================>.............] - ETA: 0s - loss: 1.4831 - accuracy: 0.4798 951/1563 [=================>............] - ETA: 0s - loss: 1.4837 - accuracy: 0.4794 989/1563 [=================>............] - ETA: 0s - loss: 1.4831 - accuracy: 0.4788 1027/1563 [==================>...........] - ETA: 0s - loss: 1.4828 - accuracy: 0.4787 1065/1563 [===================>..........] - ETA: 0s - loss: 1.4837 - accuracy: 0.4781 1103/1563 [====================>.........] - ETA: 0s - loss: 1.4832 - accuracy: 0.4787 1141/1563 [====================>.........] - ETA: 0s - loss: 1.4811 - accuracy: 0.4798 1179/1563 [=====================>........] - ETA: 0s - loss: 1.4803 - accuracy: 0.4800 1217/1563 [======================>.......] - ETA: 0s - loss: 1.4806 - accuracy: 0.4796 1255/1563 [=======================>......] - ETA: 0s - loss: 1.4808 - accuracy: 0.4794 1293/1563 [=======================>......] - ETA: 0s - loss: 1.4807 - accuracy: 0.4795 1331/1563 [========================>.....] - ETA: 0s - loss: 1.4810 - accuracy: 0.4798 1369/1563 [=========================>....] - ETA: 0s - loss: 1.4821 - accuracy: 0.4797 1407/1563 [==========================>...] - ETA: 0s - loss: 1.4810 - accuracy: 0.4800 1445/1563 [==========================>...] - ETA: 0s - loss: 1.4809 - accuracy: 0.4805 1483/1563 [===========================>..] - ETA: 0s - loss: 1.4807 - accuracy: 0.4809 1521/1563 [============================>.] - ETA: 0s - loss: 1.4805 - accuracy: 0.4812 1559/1563 [============================>.] - ETA: 0s - loss: 1.4802 - accuracy: 0.4815 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4801 - accuracy: 0.4816 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.1359 - accuracy: 0.5938 39/1563 [..............................] - ETA: 2s - loss: 1.4010 - accuracy: 0.5184 77/1563 [>.............................] - ETA: 1s - loss: 1.4132 - accuracy: 0.5093 115/1563 [=>............................] - ETA: 1s - loss: 1.4203 - accuracy: 0.5038 153/1563 [=>............................] - ETA: 1s - loss: 1.4282 - accuracy: 0.5006 191/1563 [==>...........................] - ETA: 1s - loss: 1.4348 - accuracy: 0.4998 229/1563 [===>..........................] - ETA: 1s - loss: 1.4390 - accuracy: 0.4981 267/1563 [====>.........................] - ETA: 1s - loss: 1.4370 - accuracy: 0.4963 305/1563 [====>.........................] - ETA: 1s - loss: 1.4385 - accuracy: 0.4927 343/1563 [=====>........................] - ETA: 1s - loss: 1.4346 - accuracy: 0.4937 381/1563 [======>.......................] - ETA: 1s - loss: 1.4359 - accuracy: 0.4943 419/1563 [=======>......................] - ETA: 1s - loss: 1.4361 - accuracy: 0.4942 457/1563 [=======>......................] - ETA: 1s - loss: 1.4376 - accuracy: 0.4930 495/1563 [========>.....................] - ETA: 1s - loss: 1.4403 - accuracy: 0.4910 533/1563 [=========>....................] - ETA: 1s - loss: 1.4426 - accuracy: 0.4910 571/1563 [=========>....................] - ETA: 1s - loss: 1.4403 - accuracy: 0.4915 609/1563 [==========>...................] - ETA: 1s - loss: 1.4403 - accuracy: 0.4917 647/1563 [===========>..................] - ETA: 1s - loss: 1.4437 - accuracy: 0.4911 685/1563 [============>.................] - ETA: 1s - loss: 1.4436 - accuracy: 0.4896 723/1563 [============>.................] - ETA: 1s - loss: 1.4431 - accuracy: 0.4897 761/1563 [=============>................] - ETA: 1s - loss: 1.4417 - accuracy: 0.4900 799/1563 [==============>...............] - ETA: 1s - loss: 1.4400 - accuracy: 0.4908 837/1563 [===============>..............] - ETA: 0s - loss: 1.4415 - accuracy: 0.4903 875/1563 [===============>..............] - ETA: 0s - loss: 1.4401 - accuracy: 0.4911 913/1563 [================>.............] - ETA: 0s - loss: 1.4388 - accuracy: 0.4920 951/1563 [=================>............] - ETA: 0s - loss: 1.4380 - accuracy: 0.4924 989/1563 [=================>............] - ETA: 0s - loss: 1.4367 - accuracy: 0.4932 1026/1563 [==================>...........] - ETA: 0s - loss: 1.4360 - accuracy: 0.4940 1064/1563 [===================>..........] - ETA: 0s - loss: 1.4348 - accuracy: 0.4946 1102/1563 [====================>.........] - ETA: 0s - loss: 1.4343 - accuracy: 0.4947 1140/1563 [====================>.........] - ETA: 0s - loss: 1.4346 - accuracy: 0.4948 1178/1563 [=====================>........] - ETA: 0s - loss: 1.4332 - accuracy: 0.4948 1216/1563 [======================>.......] - ETA: 0s - loss: 1.4319 - accuracy: 0.4956 1253/1563 [=======================>......] - ETA: 0s - loss: 1.4314 - accuracy: 0.4960 1291/1563 [=======================>......] - ETA: 0s - loss: 1.4305 - accuracy: 0.4964 1329/1563 [========================>.....] - ETA: 0s - loss: 1.4306 - accuracy: 0.4966 1367/1563 [=========================>....] - ETA: 0s - loss: 1.4312 - accuracy: 0.4960 1405/1563 [=========================>....] - ETA: 0s - loss: 1.4309 - accuracy: 0.4962 1443/1563 [==========================>...] - ETA: 0s - loss: 1.4308 - accuracy: 0.4962 1481/1563 [===========================>..] - ETA: 0s - loss: 1.4308 - accuracy: 0.4968 1519/1563 [============================>.] - ETA: 0s - loss: 1.4308 - accuracy: 0.4972 1557/1563 [============================>.] - ETA: 0s - loss: 1.4309 - accuracy: 0.4973 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4312 - accuracy: 0.4973 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.4908 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.3937 - accuracy: 0.5152 76/1563 [>.............................] - ETA: 2s - loss: 1.4214 - accuracy: 0.5012 114/1563 [=>............................] - ETA: 1s - loss: 1.4098 - accuracy: 0.5052 152/1563 [=>............................] - ETA: 1s - loss: 1.4036 - accuracy: 0.5047 190/1563 [==>...........................] - ETA: 1s - loss: 1.3988 - accuracy: 0.5044 228/1563 [===>..........................] - ETA: 1s - loss: 1.3935 - accuracy: 0.5070 266/1563 [====>.........................] - ETA: 1s - loss: 1.3976 - accuracy: 0.5070 304/1563 [====>.........................] - ETA: 1s - loss: 1.3955 - accuracy: 0.5083 342/1563 [=====>........................] - ETA: 1s - loss: 1.3926 - accuracy: 0.5099 380/1563 [======>.......................] - ETA: 1s - loss: 1.3902 - accuracy: 0.5109 418/1563 [=======>......................] - ETA: 1s - loss: 1.3885 - accuracy: 0.5120 456/1563 [=======>......................] - ETA: 1s - loss: 1.3856 - accuracy: 0.5123 494/1563 [========>.....................] - ETA: 1s - loss: 1.3863 - accuracy: 0.5125 532/1563 [=========>....................] - ETA: 1s - loss: 1.3881 - accuracy: 0.5119 570/1563 [=========>....................] - ETA: 1s - loss: 1.3876 - accuracy: 0.5121 608/1563 [==========>...................] - ETA: 1s - loss: 1.3872 - accuracy: 0.5128 646/1563 [===========>..................] - ETA: 1s - loss: 1.3869 - accuracy: 0.5130 684/1563 [============>.................] - ETA: 1s - loss: 1.3894 - accuracy: 0.5118 722/1563 [============>.................] - ETA: 1s - loss: 1.3886 - accuracy: 0.5127 760/1563 [=============>................] - ETA: 1s - loss: 1.3888 - accuracy: 0.5123 798/1563 [==============>...............] - ETA: 1s - loss: 1.3899 - accuracy: 0.5121 836/1563 [===============>..............] - ETA: 0s - loss: 1.3901 - accuracy: 0.5125 874/1563 [===============>..............] - ETA: 0s - loss: 1.3906 - accuracy: 0.5121 912/1563 [================>.............] - ETA: 0s - loss: 1.3908 - accuracy: 0.5119 950/1563 [=================>............] - ETA: 0s - loss: 1.3930 - accuracy: 0.5118 988/1563 [=================>............] - ETA: 0s - loss: 1.3928 - accuracy: 0.5121 1026/1563 [==================>...........] - ETA: 0s - loss: 1.3920 - accuracy: 0.5121 1064/1563 [===================>..........] - ETA: 0s - loss: 1.3921 - accuracy: 0.5121 1102/1563 [====================>.........] - ETA: 0s - loss: 1.3920 - accuracy: 0.5120 1140/1563 [====================>.........] - ETA: 0s - loss: 1.3934 - accuracy: 0.5116 1178/1563 [=====================>........] - ETA: 0s - loss: 1.3933 - accuracy: 0.5118 1216/1563 [======================>.......] - ETA: 0s - loss: 1.3941 - accuracy: 0.5119 1254/1563 [=======================>......] - ETA: 0s - loss: 1.3925 - accuracy: 0.5122 1292/1563 [=======================>......] - ETA: 0s - loss: 1.3923 - accuracy: 0.5119 1330/1563 [========================>.....] - ETA: 0s - loss: 1.3920 - accuracy: 0.5123 1368/1563 [=========================>....] - ETA: 0s - loss: 1.3914 - accuracy: 0.5132 1406/1563 [=========================>....] - ETA: 0s - loss: 1.3924 - accuracy: 0.5132 1444/1563 [==========================>...] - ETA: 0s - loss: 1.3910 - accuracy: 0.5135 1482/1563 [===========================>..] - ETA: 0s - loss: 1.3911 - accuracy: 0.5132 1520/1563 [============================>.] - ETA: 0s - loss: 1.3905 - accuracy: 0.5132 1558/1563 [============================>.] - ETA: 0s - loss: 1.3910 - accuracy: 0.5129 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3909 - accuracy: 0.5129 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.4310 - accuracy: 0.5625 38/1563 [..............................] - ETA: 2s - loss: 1.3686 - accuracy: 0.5345 76/1563 [>.............................] - ETA: 2s - loss: 1.3487 - accuracy: 0.5395 114/1563 [=>............................] - ETA: 1s - loss: 1.3576 - accuracy: 0.5304 152/1563 [=>............................] - ETA: 1s - loss: 1.3550 - accuracy: 0.5335 190/1563 [==>...........................] - ETA: 1s - loss: 1.3532 - accuracy: 0.5336 228/1563 [===>..........................] - ETA: 1s - loss: 1.3553 - accuracy: 0.5293 266/1563 [====>.........................] - ETA: 1s - loss: 1.3497 - accuracy: 0.5307 304/1563 [====>.........................] - ETA: 1s - loss: 1.3506 - accuracy: 0.5315 342/1563 [=====>........................] - ETA: 1s - loss: 1.3484 - accuracy: 0.5319 380/1563 [======>.......................] - ETA: 1s - loss: 1.3496 - accuracy: 0.5308 418/1563 [=======>......................] - ETA: 1s - loss: 1.3496 - accuracy: 0.5295 456/1563 [=======>......................] - ETA: 1s - loss: 1.3513 - accuracy: 0.5296 494/1563 [========>.....................] - ETA: 1s - loss: 1.3489 - accuracy: 0.5297 532/1563 [=========>....................] - ETA: 1s - loss: 1.3497 - accuracy: 0.5291 570/1563 [=========>....................] - ETA: 1s - loss: 1.3516 - accuracy: 0.5287 608/1563 [==========>...................] - ETA: 1s - loss: 1.3486 - accuracy: 0.5297 645/1563 [===========>..................] - ETA: 1s - loss: 1.3463 - accuracy: 0.5305 683/1563 [============>.................] - ETA: 1s - loss: 1.3490 - accuracy: 0.5298 721/1563 [============>.................] - ETA: 1s - loss: 1.3513 - accuracy: 0.5281 759/1563 [=============>................] - ETA: 1s - loss: 1.3520 - accuracy: 0.5278 797/1563 [==============>...............] - ETA: 1s - loss: 1.3487 - accuracy: 0.5287 835/1563 [===============>..............] - ETA: 0s - loss: 1.3482 - accuracy: 0.5293 873/1563 [===============>..............] - ETA: 0s - loss: 1.3472 - accuracy: 0.5296 911/1563 [================>.............] - ETA: 0s - loss: 1.3482 - accuracy: 0.5290 948/1563 [=================>............] - ETA: 0s - loss: 1.3498 - accuracy: 0.5288 986/1563 [=================>............] - ETA: 0s - loss: 1.3484 - accuracy: 0.5299 1024/1563 [==================>...........] - ETA: 0s - loss: 1.3484 - accuracy: 0.5297 1062/1563 [===================>..........] - ETA: 0s - loss: 1.3487 - accuracy: 0.5292 1100/1563 [====================>.........] - ETA: 0s - loss: 1.3488 - accuracy: 0.5287 1137/1563 [====================>.........] - ETA: 0s - loss: 1.3490 - accuracy: 0.5286 1175/1563 [=====================>........] - ETA: 0s - loss: 1.3497 - accuracy: 0.5286 1213/1563 [======================>.......] - ETA: 0s - loss: 1.3511 - accuracy: 0.5279 1251/1563 [=======================>......] - ETA: 0s - loss: 1.3511 - accuracy: 0.5274 1289/1563 [=======================>......] - ETA: 0s - loss: 1.3513 - accuracy: 0.5271 1327/1563 [========================>.....] - ETA: 0s - loss: 1.3511 - accuracy: 0.5272 1365/1563 [=========================>....] - ETA: 0s - loss: 1.3495 - accuracy: 0.5280 1403/1563 [=========================>....] - ETA: 0s - loss: 1.3493 - accuracy: 0.5282 1441/1563 [==========================>...] - ETA: 0s - loss: 1.3482 - accuracy: 0.5288 1479/1563 [===========================>..] - ETA: 0s - loss: 1.3479 - accuracy: 0.5287 1517/1563 [============================>.] - ETA: 0s - loss: 1.3485 - accuracy: 0.5285 1555/1563 [============================>.] - ETA: 0s - loss: 1.3497 - accuracy: 0.5277 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3502 - accuracy: 0.5275 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.7188 - accuracy: 0.3750 39/1563 [..............................] - ETA: 2s - loss: 1.3162 - accuracy: 0.5361 77/1563 [>.............................] - ETA: 1s - loss: 1.3143 - accuracy: 0.5345 115/1563 [=>............................] - ETA: 1s - loss: 1.3074 - accuracy: 0.5389 153/1563 [=>............................] - ETA: 1s - loss: 1.3109 - accuracy: 0.5384 191/1563 [==>...........................] - ETA: 1s - loss: 1.3120 - accuracy: 0.5376 229/1563 [===>..........................] - ETA: 1s - loss: 1.3144 - accuracy: 0.5374 267/1563 [====>.........................] - ETA: 1s - loss: 1.3162 - accuracy: 0.5378 305/1563 [====>.........................] - ETA: 1s - loss: 1.3115 - accuracy: 0.5410 343/1563 [=====>........................] - ETA: 1s - loss: 1.3102 - accuracy: 0.5405 381/1563 [======>.......................] - ETA: 1s - loss: 1.3115 - accuracy: 0.5392 419/1563 [=======>......................] - ETA: 1s - loss: 1.3116 - accuracy: 0.5392 457/1563 [=======>......................] - ETA: 1s - loss: 1.3110 - accuracy: 0.5397 495/1563 [========>.....................] - ETA: 1s - loss: 1.3096 - accuracy: 0.5403 533/1563 [=========>....................] - ETA: 1s - loss: 1.3090 - accuracy: 0.5410 570/1563 [=========>....................] - ETA: 1s - loss: 1.3095 - accuracy: 0.5403 607/1563 [==========>...................] - ETA: 1s - loss: 1.3077 - accuracy: 0.5420 645/1563 [===========>..................] - ETA: 1s - loss: 1.3090 - accuracy: 0.5416 683/1563 [============>.................] - ETA: 1s - loss: 1.3081 - accuracy: 0.5415 721/1563 [============>.................] - ETA: 1s - loss: 1.3098 - accuracy: 0.5411 759/1563 [=============>................] - ETA: 1s - loss: 1.3103 - accuracy: 0.5403 797/1563 [==============>...............] - ETA: 1s - loss: 1.3089 - accuracy: 0.5414 835/1563 [===============>..............] - ETA: 0s - loss: 1.3105 - accuracy: 0.5402 873/1563 [===============>..............] - ETA: 0s - loss: 1.3110 - accuracy: 0.5395 911/1563 [================>.............] - ETA: 0s - loss: 1.3088 - accuracy: 0.5401 949/1563 [=================>............] - ETA: 0s - loss: 1.3075 - accuracy: 0.5409 987/1563 [=================>............] - ETA: 0s - loss: 1.3084 - accuracy: 0.5404 1025/1563 [==================>...........] - ETA: 0s - loss: 1.3085 - accuracy: 0.5401 1063/1563 [===================>..........] - ETA: 0s - loss: 1.3083 - accuracy: 0.5403 1101/1563 [====================>.........] - ETA: 0s - loss: 1.3105 - accuracy: 0.5397 1139/1563 [====================>.........] - ETA: 0s - loss: 1.3113 - accuracy: 0.5393 1177/1563 [=====================>........] - ETA: 0s - loss: 1.3127 - accuracy: 0.5387 1215/1563 [======================>.......] - ETA: 0s - loss: 1.3116 - accuracy: 0.5389 1252/1563 [=======================>......] - ETA: 0s - loss: 1.3126 - accuracy: 0.5383 1290/1563 [=======================>......] - ETA: 0s - loss: 1.3118 - accuracy: 0.5387 1328/1563 [========================>.....] - ETA: 0s - loss: 1.3119 - accuracy: 0.5388 1366/1563 [=========================>....] - ETA: 0s - loss: 1.3126 - accuracy: 0.5391 1404/1563 [=========================>....] - ETA: 0s - loss: 1.3131 - accuracy: 0.5390 1442/1563 [==========================>...] - ETA: 0s - loss: 1.3140 - accuracy: 0.5388 1479/1563 [===========================>..] - ETA: 0s - loss: 1.3131 - accuracy: 0.5394 1517/1563 [============================>.] - ETA: 0s - loss: 1.3138 - accuracy: 0.5394 1555/1563 [============================>.] - ETA: 0s - loss: 1.3136 - accuracy: 0.5392 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3133 - accuracy: 0.5392 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.3414 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.2356 - accuracy: 0.5681 77/1563 [>.............................] - ETA: 1s - loss: 1.2601 - accuracy: 0.5536 115/1563 [=>............................] - ETA: 1s - loss: 1.2561 - accuracy: 0.5568 153/1563 [=>............................] - ETA: 1s - loss: 1.2564 - accuracy: 0.5584 191/1563 [==>...........................] - ETA: 1s - loss: 1.2593 - accuracy: 0.5587 229/1563 [===>..........................] - ETA: 1s - loss: 1.2659 - accuracy: 0.5565 267/1563 [====>.........................] - ETA: 1s - loss: 1.2763 - accuracy: 0.5535 305/1563 [====>.........................] - ETA: 1s - loss: 1.2756 - accuracy: 0.5522 343/1563 [=====>........................] - ETA: 1s - loss: 1.2763 - accuracy: 0.5525 381/1563 [======>.......................] - ETA: 1s - loss: 1.2797 - accuracy: 0.5500 419/1563 [=======>......................] - ETA: 1s - loss: 1.2834 - accuracy: 0.5481 457/1563 [=======>......................] - ETA: 1s - loss: 1.2841 - accuracy: 0.5484 495/1563 [========>.....................] - ETA: 1s - loss: 1.2843 - accuracy: 0.5495 533/1563 [=========>....................] - ETA: 1s - loss: 1.2851 - accuracy: 0.5493 571/1563 [=========>....................] - ETA: 1s - loss: 1.2833 - accuracy: 0.5499 609/1563 [==========>...................] - ETA: 1s - loss: 1.2816 - accuracy: 0.5491 647/1563 [===========>..................] - ETA: 1s - loss: 1.2815 - accuracy: 0.5488 685/1563 [============>.................] - ETA: 1s - loss: 1.2833 - accuracy: 0.5480 723/1563 [============>.................] - ETA: 1s - loss: 1.2813 - accuracy: 0.5498 761/1563 [=============>................] - ETA: 1s - loss: 1.2821 - accuracy: 0.5499 799/1563 [==============>...............] - ETA: 1s - loss: 1.2819 - accuracy: 0.5508 837/1563 [===============>..............] - ETA: 0s - loss: 1.2824 - accuracy: 0.5500 875/1563 [===============>..............] - ETA: 0s - loss: 1.2828 - accuracy: 0.5496 912/1563 [================>.............] - ETA: 0s - loss: 1.2810 - accuracy: 0.5508 950/1563 [=================>............] - ETA: 0s - loss: 1.2819 - accuracy: 0.5503 988/1563 [=================>............] - ETA: 0s - loss: 1.2806 - accuracy: 0.5508 1026/1563 [==================>...........] - ETA: 0s - loss: 1.2810 - accuracy: 0.5507 1064/1563 [===================>..........] - ETA: 0s - loss: 1.2804 - accuracy: 0.5508 1101/1563 [====================>.........] - ETA: 0s - loss: 1.2809 - accuracy: 0.5505 1139/1563 [====================>.........] - ETA: 0s - loss: 1.2791 - accuracy: 0.5514 1177/1563 [=====================>........] - ETA: 0s - loss: 1.2782 - accuracy: 0.5515 1214/1563 [======================>.......] - ETA: 0s - loss: 1.2786 - accuracy: 0.5517 1252/1563 [=======================>......] - ETA: 0s - loss: 1.2788 - accuracy: 0.5518 1290/1563 [=======================>......] - ETA: 0s - loss: 1.2793 - accuracy: 0.5512 1328/1563 [========================>.....] - ETA: 0s - loss: 1.2807 - accuracy: 0.5507 1366/1563 [=========================>....] - ETA: 0s - loss: 1.2811 - accuracy: 0.5506 1403/1563 [=========================>....] - ETA: 0s - loss: 1.2811 - accuracy: 0.5504 1441/1563 [==========================>...] - ETA: 0s - loss: 1.2819 - accuracy: 0.5500 1478/1563 [===========================>..] - ETA: 0s - loss: 1.2818 - accuracy: 0.5502 1516/1563 [============================>.] - ETA: 0s - loss: 1.2813 - accuracy: 0.5506 1554/1563 [============================>.] - ETA: 0s - loss: 1.2813 - accuracy: 0.5505 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2812 - accuracy: 0.5503 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.0484 - accuracy: 0.6250 39/1563 [..............................] - ETA: 2s - loss: 1.2836 - accuracy: 0.5633 77/1563 [>.............................] - ETA: 1s - loss: 1.2490 - accuracy: 0.5767 115/1563 [=>............................] - ETA: 1s - loss: 1.2608 - accuracy: 0.5679 153/1563 [=>............................] - ETA: 1s - loss: 1.2685 - accuracy: 0.5623 191/1563 [==>...........................] - ETA: 1s - loss: 1.2700 - accuracy: 0.5607 229/1563 [===>..........................] - ETA: 1s - loss: 1.2683 - accuracy: 0.5635 267/1563 [====>.........................] - ETA: 1s - loss: 1.2649 - accuracy: 0.5641 305/1563 [====>.........................] - ETA: 1s - loss: 1.2522 - accuracy: 0.5690 343/1563 [=====>........................] - ETA: 1s - loss: 1.2523 - accuracy: 0.5688 381/1563 [======>.......................] - ETA: 1s - loss: 1.2545 - accuracy: 0.5676 419/1563 [=======>......................] - ETA: 1s - loss: 1.2557 - accuracy: 0.5660 457/1563 [=======>......................] - ETA: 1s - loss: 1.2561 - accuracy: 0.5643 495/1563 [========>.....................] - ETA: 1s - loss: 1.2523 - accuracy: 0.5648 533/1563 [=========>....................] - ETA: 1s - loss: 1.2504 - accuracy: 0.5655 571/1563 [=========>....................] - ETA: 1s - loss: 1.2524 - accuracy: 0.5639 609/1563 [==========>...................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5635 647/1563 [===========>..................] - ETA: 1s - loss: 1.2517 - accuracy: 0.5639 685/1563 [============>.................] - ETA: 1s - loss: 1.2517 - accuracy: 0.5630 723/1563 [============>.................] - ETA: 1s - loss: 1.2527 - accuracy: 0.5625 761/1563 [=============>................] - ETA: 1s - loss: 1.2553 - accuracy: 0.5604 799/1563 [==============>...............] - ETA: 1s - loss: 1.2538 - accuracy: 0.5612 837/1563 [===============>..............] - ETA: 0s - loss: 1.2525 - accuracy: 0.5617 875/1563 [===============>..............] - ETA: 0s - loss: 1.2510 - accuracy: 0.5624 913/1563 [================>.............] - ETA: 0s - loss: 1.2501 - accuracy: 0.5621 951/1563 [=================>............] - ETA: 0s - loss: 1.2506 - accuracy: 0.5620 989/1563 [=================>............] - ETA: 0s - loss: 1.2498 - accuracy: 0.5623 1027/1563 [==================>...........] - ETA: 0s - loss: 1.2519 - accuracy: 0.5620 1065/1563 [===================>..........] - ETA: 0s - loss: 1.2518 - accuracy: 0.5617 1103/1563 [====================>.........] - ETA: 0s - loss: 1.2521 - accuracy: 0.5620 1141/1563 [====================>.........] - ETA: 0s - loss: 1.2509 - accuracy: 0.5624 1179/1563 [=====================>........] - ETA: 0s - loss: 1.2519 - accuracy: 0.5622 1217/1563 [======================>.......] - ETA: 0s - loss: 1.2522 - accuracy: 0.5617 1255/1563 [=======================>......] - ETA: 0s - loss: 1.2514 - accuracy: 0.5620 1293/1563 [=======================>......] - ETA: 0s - loss: 1.2514 - accuracy: 0.5617 1331/1563 [========================>.....] - ETA: 0s - loss: 1.2522 - accuracy: 0.5611 1369/1563 [=========================>....] - ETA: 0s - loss: 1.2529 - accuracy: 0.5609 1407/1563 [==========================>...] - ETA: 0s - loss: 1.2517 - accuracy: 0.5612 1445/1563 [==========================>...] - ETA: 0s - loss: 1.2516 - accuracy: 0.5614 1482/1563 [===========================>..] - ETA: 0s - loss: 1.2527 - accuracy: 0.5609 1520/1563 [============================>.] - ETA: 0s - loss: 1.2524 - accuracy: 0.5608 1558/1563 [============================>.] - ETA: 0s - loss: 1.2524 - accuracy: 0.5607 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2523 - accuracy: 0.5608 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 37.12149000167847 hydro05 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins5904094894903426355.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS