Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 98986 queued and waiting for resources srun: job 98986 has been allocated resources 2024-12-02 07:43:02.929873: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-12-02 07:43:03.073253: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-12-02 07:43:04.070692: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-12-02 07:43:04.070765: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-12-02 07:43:04.070771: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-12-02 07:43:07.183870: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-12-02 07:43:07.555410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-12-02 07:43:10.245987: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 28:20 - loss: 2.4187 - accuracy: 0.0625 36/1563 [..............................] - ETA: 2s - loss: 2.2740 - accuracy: 0.1510  72/1563 [>.............................] - ETA: 2s - loss: 2.1964 - accuracy: 0.1901 109/1563 [=>............................] - ETA: 2s - loss: 2.1414 - accuracy: 0.2139 146/1563 [=>............................] - ETA: 1s - loss: 2.1086 - accuracy: 0.2318 183/1563 [==>...........................] - ETA: 1s - loss: 2.0829 - accuracy: 0.2469 220/1563 [===>..........................] - ETA: 1s - loss: 2.0543 - accuracy: 0.2553 257/1563 [===>..........................] - ETA: 1s - loss: 2.0374 - accuracy: 0.2611 295/1563 [====>.........................] - ETA: 1s - loss: 2.0206 - accuracy: 0.2671 332/1563 [=====>........................] - ETA: 1s - loss: 2.0055 - accuracy: 0.2747 370/1563 [======>.......................] - ETA: 1s - loss: 1.9915 - accuracy: 0.2793 408/1563 [======>.......................] - ETA: 1s - loss: 1.9762 - accuracy: 0.2849 445/1563 [=======>......................] - ETA: 1s - loss: 1.9661 - accuracy: 0.2892 482/1563 [========>.....................] - ETA: 1s - loss: 1.9589 - accuracy: 0.2931 520/1563 [========>.....................] - ETA: 1s - loss: 1.9500 - accuracy: 0.2973 558/1563 [=========>....................] - ETA: 1s - loss: 1.9415 - accuracy: 0.3023 596/1563 [==========>...................] - ETA: 1s - loss: 1.9331 - accuracy: 0.3053 634/1563 [===========>..................] - ETA: 1s - loss: 1.9245 - accuracy: 0.3086 672/1563 [===========>..................] - ETA: 1s - loss: 1.9183 - accuracy: 0.3118 710/1563 [============>.................] - ETA: 1s - loss: 1.9127 - accuracy: 0.3149 748/1563 [=============>................] - ETA: 1s - loss: 1.9047 - accuracy: 0.3186 786/1563 [==============>...............] - ETA: 1s - loss: 1.8981 - accuracy: 0.3216 824/1563 [==============>...............] - ETA: 1s - loss: 1.8944 - accuracy: 0.3230 862/1563 [===============>..............] - ETA: 0s - loss: 1.8887 - accuracy: 0.3252 900/1563 [================>.............] - ETA: 0s - loss: 1.8848 - accuracy: 0.3264 938/1563 [=================>............] - ETA: 0s - loss: 1.8780 - accuracy: 0.3292 976/1563 [=================>............] - ETA: 0s - loss: 1.8731 - accuracy: 0.3306 1014/1563 [==================>...........] - ETA: 0s - loss: 1.8674 - accuracy: 0.3323 1052/1563 [===================>..........] - ETA: 0s - loss: 1.8617 - accuracy: 0.3354 1090/1563 [===================>..........] - ETA: 0s - loss: 1.8572 - accuracy: 0.3372 1128/1563 [====================>.........] - ETA: 0s - loss: 1.8523 - accuracy: 0.3393 1166/1563 [=====================>........] - ETA: 0s - loss: 1.8484 - accuracy: 0.3411 1203/1563 [======================>.......] - ETA: 0s - loss: 1.8441 - accuracy: 0.3429 1241/1563 [======================>.......] - ETA: 0s - loss: 1.8402 - accuracy: 0.3439 1279/1563 [=======================>......] - ETA: 0s - loss: 1.8368 - accuracy: 0.3450 1317/1563 [========================>.....] - ETA: 0s - loss: 1.8335 - accuracy: 0.3459 1355/1563 [=========================>....] - ETA: 0s - loss: 1.8299 - accuracy: 0.3474 1393/1563 [=========================>....] - ETA: 0s - loss: 1.8271 - accuracy: 0.3485 1431/1563 [==========================>...] - ETA: 0s - loss: 1.8246 - accuracy: 0.3494 1469/1563 [===========================>..] - ETA: 0s - loss: 1.8210 - accuracy: 0.3512 1507/1563 [===========================>..] - ETA: 0s - loss: 1.8179 - accuracy: 0.3524 1544/1563 [============================>.] - ETA: 0s - loss: 1.8157 - accuracy: 0.3533 1563/1563 [==============================] - 3s 1ms/step - loss: 1.8139 - accuracy: 0.3540 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.6694 - accuracy: 0.5000 38/1563 [..............................] - ETA: 2s - loss: 1.7148 - accuracy: 0.3939 76/1563 [>.............................] - ETA: 2s - loss: 1.6911 - accuracy: 0.4034 114/1563 [=>............................] - ETA: 1s - loss: 1.6803 - accuracy: 0.4084 152/1563 [=>............................] - ETA: 1s - loss: 1.6733 - accuracy: 0.4102 190/1563 [==>...........................] - ETA: 1s - loss: 1.6713 - accuracy: 0.4130 228/1563 [===>..........................] - ETA: 1s - loss: 1.6676 - accuracy: 0.4134 262/1563 [====>.........................] - ETA: 1s - loss: 1.6606 - accuracy: 0.4167 300/1563 [====>.........................] - ETA: 1s - loss: 1.6619 - accuracy: 0.4155 338/1563 [=====>........................] - ETA: 1s - loss: 1.6607 - accuracy: 0.4162 376/1563 [======>.......................] - ETA: 1s - loss: 1.6578 - accuracy: 0.4179 414/1563 [======>.......................] - ETA: 1s - loss: 1.6544 - accuracy: 0.4189 452/1563 [=======>......................] - ETA: 1s - loss: 1.6517 - accuracy: 0.4195 490/1563 [========>.....................] - ETA: 1s - loss: 1.6514 - accuracy: 0.4188 528/1563 [=========>....................] - ETA: 1s - loss: 1.6518 - accuracy: 0.4176 565/1563 [=========>....................] - ETA: 1s - loss: 1.6495 - accuracy: 0.4183 603/1563 [==========>...................] - ETA: 1s - loss: 1.6444 - accuracy: 0.4184 641/1563 [===========>..................] - ETA: 1s - loss: 1.6432 - accuracy: 0.4194 678/1563 [============>.................] - ETA: 1s - loss: 1.6405 - accuracy: 0.4203 716/1563 [============>.................] - ETA: 1s - loss: 1.6405 - accuracy: 0.4205 754/1563 [=============>................] - ETA: 1s - loss: 1.6374 - accuracy: 0.4215 792/1563 [==============>...............] - ETA: 1s - loss: 1.6360 - accuracy: 0.4219 830/1563 [==============>...............] - ETA: 0s - loss: 1.6374 - accuracy: 0.4219 868/1563 [===============>..............] - ETA: 0s - loss: 1.6375 - accuracy: 0.4219 906/1563 [================>.............] - ETA: 0s - loss: 1.6374 - accuracy: 0.4219 944/1563 [=================>............] - ETA: 0s - loss: 1.6369 - accuracy: 0.4221 982/1563 [=================>............] - ETA: 0s - loss: 1.6352 - accuracy: 0.4230 1019/1563 [==================>...........] - ETA: 0s - loss: 1.6334 - accuracy: 0.4239 1057/1563 [===================>..........] - ETA: 0s - loss: 1.6308 - accuracy: 0.4248 1095/1563 [====================>.........] - ETA: 0s - loss: 1.6289 - accuracy: 0.4251 1133/1563 [====================>.........] - ETA: 0s - loss: 1.6276 - accuracy: 0.4263 1171/1563 [=====================>........] - ETA: 0s - loss: 1.6283 - accuracy: 0.4264 1208/1563 [======================>.......] - ETA: 0s - loss: 1.6277 - accuracy: 0.4272 1245/1563 [======================>.......] - ETA: 0s - loss: 1.6275 - accuracy: 0.4274 1282/1563 [=======================>......] - ETA: 0s - loss: 1.6259 - accuracy: 0.4278 1319/1563 [========================>.....] - ETA: 0s - loss: 1.6254 - accuracy: 0.4280 1357/1563 [=========================>....] - ETA: 0s - loss: 1.6246 - accuracy: 0.4280 1394/1563 [=========================>....] - ETA: 0s - loss: 1.6236 - accuracy: 0.4280 1431/1563 [==========================>...] - ETA: 0s - loss: 1.6237 - accuracy: 0.4281 1468/1563 [===========================>..] - ETA: 0s - loss: 1.6231 - accuracy: 0.4285 1505/1563 [===========================>..] - ETA: 0s - loss: 1.6229 - accuracy: 0.4283 1542/1563 [============================>.] - ETA: 0s - loss: 1.6232 - accuracy: 0.4273 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6223 - accuracy: 0.4278 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.5167 - accuracy: 0.5000 38/1563 [..............................] - ETA: 2s - loss: 1.5552 - accuracy: 0.4564 76/1563 [>.............................] - ETA: 2s - loss: 1.5546 - accuracy: 0.4576 113/1563 [=>............................] - ETA: 1s - loss: 1.5437 - accuracy: 0.4616 151/1563 [=>............................] - ETA: 1s - loss: 1.5427 - accuracy: 0.4572 189/1563 [==>...........................] - ETA: 1s - loss: 1.5515 - accuracy: 0.4544 227/1563 [===>..........................] - ETA: 1s - loss: 1.5545 - accuracy: 0.4521 265/1563 [====>.........................] - ETA: 1s - loss: 1.5606 - accuracy: 0.4489 303/1563 [====>.........................] - ETA: 1s - loss: 1.5609 - accuracy: 0.4483 341/1563 [=====>........................] - ETA: 1s - loss: 1.5624 - accuracy: 0.4486 379/1563 [======>.......................] - ETA: 1s - loss: 1.5579 - accuracy: 0.4500 417/1563 [=======>......................] - ETA: 1s - loss: 1.5576 - accuracy: 0.4496 454/1563 [=======>......................] - ETA: 1s - loss: 1.5561 - accuracy: 0.4501 492/1563 [========>.....................] - ETA: 1s - loss: 1.5544 - accuracy: 0.4497 530/1563 [=========>....................] - ETA: 1s - loss: 1.5548 - accuracy: 0.4491 568/1563 [=========>....................] - ETA: 1s - loss: 1.5535 - accuracy: 0.4494 606/1563 [==========>...................] - ETA: 1s - loss: 1.5515 - accuracy: 0.4496 644/1563 [===========>..................] - ETA: 1s - loss: 1.5510 - accuracy: 0.4503 682/1563 [============>.................] - ETA: 1s - loss: 1.5514 - accuracy: 0.4503 720/1563 [============>.................] - ETA: 1s - loss: 1.5500 - accuracy: 0.4521 757/1563 [=============>................] - ETA: 1s - loss: 1.5500 - accuracy: 0.4518 795/1563 [==============>...............] - ETA: 1s - loss: 1.5518 - accuracy: 0.4506 833/1563 [==============>...............] - ETA: 0s - loss: 1.5527 - accuracy: 0.4514 871/1563 [===============>..............] - ETA: 0s - loss: 1.5537 - accuracy: 0.4512 909/1563 [================>.............] - ETA: 0s - loss: 1.5506 - accuracy: 0.4524 947/1563 [=================>............] - ETA: 0s - loss: 1.5486 - accuracy: 0.4534 985/1563 [=================>............] - ETA: 0s - loss: 1.5477 - accuracy: 0.4540 1023/1563 [==================>...........] - ETA: 0s - loss: 1.5448 - accuracy: 0.4553 1061/1563 [===================>..........] - ETA: 0s - loss: 1.5461 - accuracy: 0.4549 1099/1563 [====================>.........] - ETA: 0s - loss: 1.5449 - accuracy: 0.4552 1137/1563 [====================>.........] - ETA: 0s - loss: 1.5465 - accuracy: 0.4549 1175/1563 [=====================>........] - ETA: 0s - loss: 1.5462 - accuracy: 0.4544 1213/1563 [======================>.......] - ETA: 0s - loss: 1.5465 - accuracy: 0.4538 1251/1563 [=======================>......] - ETA: 0s - loss: 1.5470 - accuracy: 0.4539 1286/1563 [=======================>......] - ETA: 0s - loss: 1.5464 - accuracy: 0.4544 1324/1563 [========================>.....] - ETA: 0s - loss: 1.5452 - accuracy: 0.4554 1362/1563 [=========================>....] - ETA: 0s - loss: 1.5445 - accuracy: 0.4555 1400/1563 [=========================>....] - ETA: 0s - loss: 1.5426 - accuracy: 0.4564 1438/1563 [==========================>...] - ETA: 0s - loss: 1.5426 - accuracy: 0.4569 1476/1563 [===========================>..] - ETA: 0s - loss: 1.5427 - accuracy: 0.4568 1514/1563 [============================>.] - ETA: 0s - loss: 1.5416 - accuracy: 0.4569 1551/1563 [============================>.] - ETA: 0s - loss: 1.5399 - accuracy: 0.4574 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5403 - accuracy: 0.4574 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.5438 - accuracy: 0.3438 36/1563 [..............................] - ETA: 2s - loss: 1.4720 - accuracy: 0.4566 74/1563 [>.............................] - ETA: 2s - loss: 1.4931 - accuracy: 0.4624 112/1563 [=>............................] - ETA: 1s - loss: 1.4840 - accuracy: 0.4718 150/1563 [=>............................] - ETA: 1s - loss: 1.4985 - accuracy: 0.4715 188/1563 [==>...........................] - ETA: 1s - loss: 1.4936 - accuracy: 0.4737 226/1563 [===>..........................] - ETA: 1s - loss: 1.4855 - accuracy: 0.4773 264/1563 [====>.........................] - ETA: 1s - loss: 1.4879 - accuracy: 0.4743 302/1563 [====>.........................] - ETA: 1s - loss: 1.4857 - accuracy: 0.4759 339/1563 [=====>........................] - ETA: 1s - loss: 1.4794 - accuracy: 0.4787 377/1563 [======>.......................] - ETA: 1s - loss: 1.4842 - accuracy: 0.4765 415/1563 [======>.......................] - ETA: 1s - loss: 1.4816 - accuracy: 0.4782 453/1563 [=======>......................] - ETA: 1s - loss: 1.4845 - accuracy: 0.4770 491/1563 [========>.....................] - ETA: 1s - loss: 1.4874 - accuracy: 0.4770 529/1563 [=========>....................] - ETA: 1s - loss: 1.4881 - accuracy: 0.4754 567/1563 [=========>....................] - ETA: 1s - loss: 1.4896 - accuracy: 0.4750 605/1563 [==========>...................] - ETA: 1s - loss: 1.4873 - accuracy: 0.4747 643/1563 [===========>..................] - ETA: 1s - loss: 1.4869 - accuracy: 0.4737 681/1563 [============>.................] - ETA: 1s - loss: 1.4892 - accuracy: 0.4724 719/1563 [============>.................] - ETA: 1s - loss: 1.4877 - accuracy: 0.4734 757/1563 [=============>................] - ETA: 1s - loss: 1.4860 - accuracy: 0.4740 795/1563 [==============>...............] - ETA: 1s - loss: 1.4862 - accuracy: 0.4739 833/1563 [==============>...............] - ETA: 0s - loss: 1.4845 - accuracy: 0.4745 871/1563 [===============>..............] - ETA: 0s - loss: 1.4852 - accuracy: 0.4738 908/1563 [================>.............] - ETA: 0s - loss: 1.4853 - accuracy: 0.4737 946/1563 [=================>............] - ETA: 0s - loss: 1.4844 - accuracy: 0.4745 984/1563 [=================>............] - ETA: 0s - loss: 1.4843 - accuracy: 0.4749 1022/1563 [==================>...........] - ETA: 0s - loss: 1.4841 - accuracy: 0.4746 1057/1563 [===================>..........] - ETA: 0s - loss: 1.4842 - accuracy: 0.4747 1095/1563 [====================>.........] - ETA: 0s - loss: 1.4848 - accuracy: 0.4740 1133/1563 [====================>.........] - ETA: 0s - loss: 1.4843 - accuracy: 0.4738 1171/1563 [=====================>........] - ETA: 0s - loss: 1.4839 - accuracy: 0.4745 1209/1563 [======================>.......] - ETA: 0s - loss: 1.4829 - accuracy: 0.4749 1246/1563 [======================>.......] - ETA: 0s - loss: 1.4819 - accuracy: 0.4755 1284/1563 [=======================>......] - ETA: 0s - loss: 1.4811 - accuracy: 0.4755 1322/1563 [========================>.....] - ETA: 0s - loss: 1.4804 - accuracy: 0.4756 1360/1563 [=========================>....] - ETA: 0s - loss: 1.4806 - accuracy: 0.4757 1398/1563 [=========================>....] - ETA: 0s - loss: 1.4818 - accuracy: 0.4755 1436/1563 [==========================>...] - ETA: 0s - loss: 1.4819 - accuracy: 0.4756 1474/1563 [===========================>..] - ETA: 0s - loss: 1.4815 - accuracy: 0.4756 1511/1563 [============================>.] - ETA: 0s - loss: 1.4816 - accuracy: 0.4756 1549/1563 [============================>.] - ETA: 0s - loss: 1.4813 - accuracy: 0.4756 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4819 - accuracy: 0.4752 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.6401 - accuracy: 0.3750 39/1563 [..............................] - ETA: 2s - loss: 1.4557 - accuracy: 0.4824 77/1563 [>.............................] - ETA: 1s - loss: 1.4410 - accuracy: 0.4931 115/1563 [=>............................] - ETA: 1s - loss: 1.4541 - accuracy: 0.4864 153/1563 [=>............................] - ETA: 1s - loss: 1.4548 - accuracy: 0.4867 191/1563 [==>...........................] - ETA: 1s - loss: 1.4582 - accuracy: 0.4846 229/1563 [===>..........................] - ETA: 1s - loss: 1.4683 - accuracy: 0.4797 267/1563 [====>.........................] - ETA: 1s - loss: 1.4644 - accuracy: 0.4799 302/1563 [====>.........................] - ETA: 1s - loss: 1.4587 - accuracy: 0.4829 340/1563 [=====>........................] - ETA: 1s - loss: 1.4594 - accuracy: 0.4834 378/1563 [======>.......................] - ETA: 1s - loss: 1.4572 - accuracy: 0.4839 416/1563 [======>.......................] - ETA: 1s - loss: 1.4545 - accuracy: 0.4831 454/1563 [=======>......................] - ETA: 1s - loss: 1.4540 - accuracy: 0.4839 492/1563 [========>.....................] - ETA: 1s - loss: 1.4546 - accuracy: 0.4832 530/1563 [=========>....................] - ETA: 1s - loss: 1.4487 - accuracy: 0.4857 567/1563 [=========>....................] - ETA: 1s - loss: 1.4471 - accuracy: 0.4857 605/1563 [==========>...................] - ETA: 1s - loss: 1.4441 - accuracy: 0.4865 643/1563 [===========>..................] - ETA: 1s - loss: 1.4432 - accuracy: 0.4864 681/1563 [============>.................] - ETA: 1s - loss: 1.4416 - accuracy: 0.4862 719/1563 [============>.................] - ETA: 1s - loss: 1.4405 - accuracy: 0.4873 757/1563 [=============>................] - ETA: 1s - loss: 1.4398 - accuracy: 0.4872 795/1563 [==============>...............] - ETA: 1s - loss: 1.4413 - accuracy: 0.4876 833/1563 [==============>...............] - ETA: 0s - loss: 1.4404 - accuracy: 0.4884 871/1563 [===============>..............] - ETA: 0s - loss: 1.4409 - accuracy: 0.4882 909/1563 [================>.............] - ETA: 0s - loss: 1.4394 - accuracy: 0.4891 946/1563 [=================>............] - ETA: 0s - loss: 1.4402 - accuracy: 0.4884 984/1563 [=================>............] - ETA: 0s - loss: 1.4411 - accuracy: 0.4888 1022/1563 [==================>...........] - ETA: 0s - loss: 1.4397 - accuracy: 0.4899 1060/1563 [===================>..........] - ETA: 0s - loss: 1.4401 - accuracy: 0.4898 1099/1563 [====================>.........] - ETA: 0s - loss: 1.4395 - accuracy: 0.4899 1137/1563 [====================>.........] - ETA: 0s - loss: 1.4383 - accuracy: 0.4904 1175/1563 [=====================>........] - ETA: 0s - loss: 1.4372 - accuracy: 0.4908 1213/1563 [======================>.......] - ETA: 0s - loss: 1.4362 - accuracy: 0.4919 1251/1563 [=======================>......] - ETA: 0s - loss: 1.4360 - accuracy: 0.4921 1288/1563 [=======================>......] - ETA: 0s - loss: 1.4349 - accuracy: 0.4925 1326/1563 [========================>.....] - ETA: 0s - loss: 1.4342 - accuracy: 0.4927 1362/1563 [=========================>....] - ETA: 0s - loss: 1.4343 - accuracy: 0.4929 1400/1563 [=========================>....] - ETA: 0s - loss: 1.4342 - accuracy: 0.4926 1439/1563 [==========================>...] - ETA: 0s - loss: 1.4344 - accuracy: 0.4924 1477/1563 [===========================>..] - ETA: 0s - loss: 1.4341 - accuracy: 0.4926 1515/1563 [============================>.] - ETA: 0s - loss: 1.4333 - accuracy: 0.4923 1553/1563 [============================>.] - ETA: 0s - loss: 1.4330 - accuracy: 0.4926 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4326 - accuracy: 0.4927 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.3868 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.4050 - accuracy: 0.5152 77/1563 [>.............................] - ETA: 1s - loss: 1.4021 - accuracy: 0.5130 115/1563 [=>............................] - ETA: 1s - loss: 1.4012 - accuracy: 0.5136 153/1563 [=>............................] - ETA: 1s - loss: 1.4048 - accuracy: 0.5123 191/1563 [==>...........................] - ETA: 1s - loss: 1.3956 - accuracy: 0.5137 229/1563 [===>..........................] - ETA: 1s - loss: 1.3912 - accuracy: 0.5142 267/1563 [====>.........................] - ETA: 1s - loss: 1.3862 - accuracy: 0.5144 304/1563 [====>.........................] - ETA: 1s - loss: 1.3900 - accuracy: 0.5121 342/1563 [=====>........................] - ETA: 1s - loss: 1.3843 - accuracy: 0.5148 380/1563 [======>.......................] - ETA: 1s - loss: 1.3868 - accuracy: 0.5137 418/1563 [=======>......................] - ETA: 1s - loss: 1.3878 - accuracy: 0.5143 456/1563 [=======>......................] - ETA: 1s - loss: 1.3924 - accuracy: 0.5119 494/1563 [========>.....................] - ETA: 1s - loss: 1.3950 - accuracy: 0.5130 532/1563 [=========>....................] - ETA: 1s - loss: 1.3956 - accuracy: 0.5127 570/1563 [=========>....................] - ETA: 1s - loss: 1.3950 - accuracy: 0.5132 608/1563 [==========>...................] - ETA: 1s - loss: 1.3972 - accuracy: 0.5125 646/1563 [===========>..................] - ETA: 1s - loss: 1.3970 - accuracy: 0.5121 684/1563 [============>.................] - ETA: 1s - loss: 1.3962 - accuracy: 0.5127 722/1563 [============>.................] - ETA: 1s - loss: 1.3964 - accuracy: 0.5135 760/1563 [=============>................] - ETA: 1s - loss: 1.3980 - accuracy: 0.5118 798/1563 [==============>...............] - ETA: 1s - loss: 1.3982 - accuracy: 0.5122 836/1563 [===============>..............] - ETA: 0s - loss: 1.3980 - accuracy: 0.5114 874/1563 [===============>..............] - ETA: 0s - loss: 1.3987 - accuracy: 0.5111 912/1563 [================>.............] - ETA: 0s - loss: 1.3985 - accuracy: 0.5113 950/1563 [=================>............] - ETA: 0s - loss: 1.3993 - accuracy: 0.5104 988/1563 [=================>............] - ETA: 0s - loss: 1.3989 - accuracy: 0.5109 1026/1563 [==================>...........] - ETA: 0s - loss: 1.3968 - accuracy: 0.5119 1064/1563 [===================>..........] - ETA: 0s - loss: 1.3969 - accuracy: 0.5120 1102/1563 [====================>.........] - ETA: 0s - loss: 1.3969 - accuracy: 0.5121 1140/1563 [====================>.........] - ETA: 0s - loss: 1.3952 - accuracy: 0.5127 1178/1563 [=====================>........] - ETA: 0s - loss: 1.3936 - accuracy: 0.5132 1215/1563 [======================>.......] - ETA: 0s - loss: 1.3937 - accuracy: 0.5130 1253/1563 [=======================>......] - ETA: 0s - loss: 1.3925 - accuracy: 0.5136 1291/1563 [=======================>......] - ETA: 0s - loss: 1.3924 - accuracy: 0.5136 1329/1563 [========================>.....] - ETA: 0s - loss: 1.3914 - accuracy: 0.5142 1367/1563 [=========================>....] - ETA: 0s - loss: 1.3906 - accuracy: 0.5140 1405/1563 [=========================>....] - ETA: 0s - loss: 1.3896 - accuracy: 0.5139 1443/1563 [==========================>...] - ETA: 0s - loss: 1.3906 - accuracy: 0.5129 1481/1563 [===========================>..] - ETA: 0s - loss: 1.3906 - accuracy: 0.5133 1519/1563 [============================>.] - ETA: 0s - loss: 1.3904 - accuracy: 0.5136 1557/1563 [============================>.] - ETA: 0s - loss: 1.3895 - accuracy: 0.5139 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3892 - accuracy: 0.5140 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.0715 - accuracy: 0.5625 39/1563 [..............................] - ETA: 2s - loss: 1.3698 - accuracy: 0.5441 77/1563 [>.............................] - ETA: 2s - loss: 1.3869 - accuracy: 0.5183 115/1563 [=>............................] - ETA: 1s - loss: 1.3663 - accuracy: 0.5207 153/1563 [=>............................] - ETA: 1s - loss: 1.3707 - accuracy: 0.5229 191/1563 [==>...........................] - ETA: 1s - loss: 1.3762 - accuracy: 0.5167 229/1563 [===>..........................] - ETA: 1s - loss: 1.3805 - accuracy: 0.5145 266/1563 [====>.........................] - ETA: 1s - loss: 1.3728 - accuracy: 0.5166 303/1563 [====>.........................] - ETA: 1s - loss: 1.3696 - accuracy: 0.5184 341/1563 [=====>........................] - ETA: 1s - loss: 1.3716 - accuracy: 0.5181 379/1563 [======>.......................] - ETA: 1s - loss: 1.3663 - accuracy: 0.5217 417/1563 [=======>......................] - ETA: 1s - loss: 1.3652 - accuracy: 0.5222 455/1563 [=======>......................] - ETA: 1s - loss: 1.3635 - accuracy: 0.5227 493/1563 [========>.....................] - ETA: 1s - loss: 1.3639 - accuracy: 0.5206 531/1563 [=========>....................] - ETA: 1s - loss: 1.3628 - accuracy: 0.5224 569/1563 [=========>....................] - ETA: 1s - loss: 1.3609 - accuracy: 0.5224 606/1563 [==========>...................] - ETA: 1s - loss: 1.3600 - accuracy: 0.5237 644/1563 [===========>..................] - ETA: 1s - loss: 1.3601 - accuracy: 0.5236 682/1563 [============>.................] - ETA: 1s - loss: 1.3610 - accuracy: 0.5233 720/1563 [============>.................] - ETA: 1s - loss: 1.3605 - accuracy: 0.5230 758/1563 [=============>................] - ETA: 1s - loss: 1.3578 - accuracy: 0.5235 796/1563 [==============>...............] - ETA: 1s - loss: 1.3573 - accuracy: 0.5245 834/1563 [===============>..............] - ETA: 0s - loss: 1.3579 - accuracy: 0.5250 872/1563 [===============>..............] - ETA: 0s - loss: 1.3574 - accuracy: 0.5249 911/1563 [================>.............] - ETA: 0s - loss: 1.3570 - accuracy: 0.5251 949/1563 [=================>............] - ETA: 0s - loss: 1.3588 - accuracy: 0.5251 987/1563 [=================>............] - ETA: 0s - loss: 1.3576 - accuracy: 0.5256 1025/1563 [==================>...........] - ETA: 0s - loss: 1.3569 - accuracy: 0.5255 1063/1563 [===================>..........] - ETA: 0s - loss: 1.3573 - accuracy: 0.5252 1101/1563 [====================>.........] - ETA: 0s - loss: 1.3559 - accuracy: 0.5255 1139/1563 [====================>.........] - ETA: 0s - loss: 1.3558 - accuracy: 0.5253 1177/1563 [=====================>........] - ETA: 0s - loss: 1.3550 - accuracy: 0.5256 1215/1563 [======================>.......] - ETA: 0s - loss: 1.3546 - accuracy: 0.5262 1252/1563 [=======================>......] - ETA: 0s - loss: 1.3537 - accuracy: 0.5265 1290/1563 [=======================>......] - ETA: 0s - loss: 1.3538 - accuracy: 0.5262 1328/1563 [========================>.....] - ETA: 0s - loss: 1.3531 - accuracy: 0.5266 1366/1563 [=========================>....] - ETA: 0s - loss: 1.3523 - accuracy: 0.5269 1404/1563 [=========================>....] - ETA: 0s - loss: 1.3520 - accuracy: 0.5271 1442/1563 [==========================>...] - ETA: 0s - loss: 1.3511 - accuracy: 0.5273 1479/1563 [===========================>..] - ETA: 0s - loss: 1.3515 - accuracy: 0.5270 1517/1563 [============================>.] - ETA: 0s - loss: 1.3505 - accuracy: 0.5277 1555/1563 [============================>.] - ETA: 0s - loss: 1.3504 - accuracy: 0.5279 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3507 - accuracy: 0.5278 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.2187 - accuracy: 0.6250 38/1563 [..............................] - ETA: 2s - loss: 1.3328 - accuracy: 0.5493 76/1563 [>.............................] - ETA: 2s - loss: 1.2996 - accuracy: 0.5502 114/1563 [=>............................] - ETA: 1s - loss: 1.2970 - accuracy: 0.5504 152/1563 [=>............................] - ETA: 1s - loss: 1.3044 - accuracy: 0.5434 190/1563 [==>...........................] - ETA: 1s - loss: 1.3045 - accuracy: 0.5400 227/1563 [===>..........................] - ETA: 1s - loss: 1.3107 - accuracy: 0.5384 264/1563 [====>.........................] - ETA: 1s - loss: 1.3157 - accuracy: 0.5359 302/1563 [====>.........................] - ETA: 1s - loss: 1.3122 - accuracy: 0.5384 340/1563 [=====>........................] - ETA: 1s - loss: 1.3161 - accuracy: 0.5361 378/1563 [======>.......................] - ETA: 1s - loss: 1.3187 - accuracy: 0.5340 416/1563 [======>.......................] - ETA: 1s - loss: 1.3149 - accuracy: 0.5347 454/1563 [=======>......................] - ETA: 1s - loss: 1.3111 - accuracy: 0.5372 492/1563 [========>.....................] - ETA: 1s - loss: 1.3111 - accuracy: 0.5377 530/1563 [=========>....................] - ETA: 1s - loss: 1.3137 - accuracy: 0.5369 568/1563 [=========>....................] - ETA: 1s - loss: 1.3133 - accuracy: 0.5380 606/1563 [==========>...................] - ETA: 1s - loss: 1.3092 - accuracy: 0.5408 644/1563 [===========>..................] - ETA: 1s - loss: 1.3106 - accuracy: 0.5396 682/1563 [============>.................] - ETA: 1s - loss: 1.3094 - accuracy: 0.5400 720/1563 [============>.................] - ETA: 1s - loss: 1.3074 - accuracy: 0.5405 758/1563 [=============>................] - ETA: 1s - loss: 1.3092 - accuracy: 0.5402 796/1563 [==============>...............] - ETA: 1s - loss: 1.3092 - accuracy: 0.5402 834/1563 [===============>..............] - ETA: 0s - loss: 1.3102 - accuracy: 0.5397 872/1563 [===============>..............] - ETA: 0s - loss: 1.3101 - accuracy: 0.5396 910/1563 [================>.............] - ETA: 0s - loss: 1.3083 - accuracy: 0.5404 948/1563 [=================>............] - ETA: 0s - loss: 1.3105 - accuracy: 0.5395 986/1563 [=================>............] - ETA: 0s - loss: 1.3101 - accuracy: 0.5395 1024/1563 [==================>...........] - ETA: 0s - loss: 1.3110 - accuracy: 0.5392 1062/1563 [===================>..........] - ETA: 0s - loss: 1.3127 - accuracy: 0.5384 1099/1563 [====================>.........] - ETA: 0s - loss: 1.3134 - accuracy: 0.5380 1137/1563 [====================>.........] - ETA: 0s - loss: 1.3149 - accuracy: 0.5377 1175/1563 [=====================>........] - ETA: 0s - loss: 1.3139 - accuracy: 0.5375 1213/1563 [======================>.......] - ETA: 0s - loss: 1.3149 - accuracy: 0.5374 1251/1563 [=======================>......] - ETA: 0s - loss: 1.3151 - accuracy: 0.5371 1289/1563 [=======================>......] - ETA: 0s - loss: 1.3152 - accuracy: 0.5368 1327/1563 [========================>.....] - ETA: 0s - loss: 1.3157 - accuracy: 0.5365 1365/1563 [=========================>....] - ETA: 0s - loss: 1.3160 - accuracy: 0.5367 1403/1563 [=========================>....] - ETA: 0s - loss: 1.3176 - accuracy: 0.5361 1441/1563 [==========================>...] - ETA: 0s - loss: 1.3167 - accuracy: 0.5363 1479/1563 [===========================>..] - ETA: 0s - loss: 1.3154 - accuracy: 0.5372 1517/1563 [============================>.] - ETA: 0s - loss: 1.3152 - accuracy: 0.5375 1556/1563 [============================>.] - ETA: 0s - loss: 1.3148 - accuracy: 0.5375 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3151 - accuracy: 0.5375 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.3538 - accuracy: 0.5000 39/1563 [..............................] - ETA: 2s - loss: 1.2640 - accuracy: 0.5753 77/1563 [>.............................] - ETA: 1s - loss: 1.2819 - accuracy: 0.5576 115/1563 [=>............................] - ETA: 1s - loss: 1.2705 - accuracy: 0.5633 152/1563 [=>............................] - ETA: 1s - loss: 1.2769 - accuracy: 0.5586 190/1563 [==>...........................] - ETA: 1s - loss: 1.2657 - accuracy: 0.5630 227/1563 [===>..........................] - ETA: 1s - loss: 1.2703 - accuracy: 0.5558 265/1563 [====>.........................] - ETA: 1s - loss: 1.2668 - accuracy: 0.5560 303/1563 [====>.........................] - ETA: 1s - loss: 1.2675 - accuracy: 0.5595 341/1563 [=====>........................] - ETA: 1s - loss: 1.2755 - accuracy: 0.5556 379/1563 [======>.......................] - ETA: 1s - loss: 1.2773 - accuracy: 0.5572 417/1563 [=======>......................] - ETA: 1s - loss: 1.2775 - accuracy: 0.5570 455/1563 [=======>......................] - ETA: 1s - loss: 1.2809 - accuracy: 0.5549 492/1563 [========>.....................] - ETA: 1s - loss: 1.2815 - accuracy: 0.5551 530/1563 [=========>....................] - ETA: 1s - loss: 1.2850 - accuracy: 0.5540 568/1563 [=========>....................] - ETA: 1s - loss: 1.2864 - accuracy: 0.5524 606/1563 [==========>...................] - ETA: 1s - loss: 1.2905 - accuracy: 0.5511 644/1563 [===========>..................] - ETA: 1s - loss: 1.2880 - accuracy: 0.5516 682/1563 [============>.................] - ETA: 1s - loss: 1.2857 - accuracy: 0.5529 720/1563 [============>.................] - ETA: 1s - loss: 1.2871 - accuracy: 0.5532 757/1563 [=============>................] - ETA: 1s - loss: 1.2863 - accuracy: 0.5539 795/1563 [==============>...............] - ETA: 1s - loss: 1.2871 - accuracy: 0.5527 833/1563 [==============>...............] - ETA: 0s - loss: 1.2866 - accuracy: 0.5523 871/1563 [===============>..............] - ETA: 0s - loss: 1.2863 - accuracy: 0.5524 909/1563 [================>.............] - ETA: 0s - loss: 1.2869 - accuracy: 0.5516 947/1563 [=================>............] - ETA: 0s - loss: 1.2885 - accuracy: 0.5509 985/1563 [=================>............] - ETA: 0s - loss: 1.2886 - accuracy: 0.5506 1023/1563 [==================>...........] - ETA: 0s - loss: 1.2880 - accuracy: 0.5504 1061/1563 [===================>..........] - ETA: 0s - loss: 1.2888 - accuracy: 0.5500 1099/1563 [====================>.........] - ETA: 0s - loss: 1.2888 - accuracy: 0.5503 1137/1563 [====================>.........] - ETA: 0s - loss: 1.2877 - accuracy: 0.5499 1175/1563 [=====================>........] - ETA: 0s - loss: 1.2853 - accuracy: 0.5503 1213/1563 [======================>.......] - ETA: 0s - loss: 1.2852 - accuracy: 0.5502 1251/1563 [=======================>......] - ETA: 0s - loss: 1.2841 - accuracy: 0.5507 1289/1563 [=======================>......] - ETA: 0s - loss: 1.2855 - accuracy: 0.5502 1327/1563 [========================>.....] - ETA: 0s - loss: 1.2853 - accuracy: 0.5501 1365/1563 [=========================>....] - ETA: 0s - loss: 1.2855 - accuracy: 0.5497 1403/1563 [=========================>....] - ETA: 0s - loss: 1.2851 - accuracy: 0.5502 1441/1563 [==========================>...] - ETA: 0s - loss: 1.2856 - accuracy: 0.5501 1479/1563 [===========================>..] - ETA: 0s - loss: 1.2853 - accuracy: 0.5498 1517/1563 [============================>.] - ETA: 0s - loss: 1.2856 - accuracy: 0.5499 1555/1563 [============================>.] - ETA: 0s - loss: 1.2853 - accuracy: 0.5504 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2856 - accuracy: 0.5504 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.5753 - accuracy: 0.4688 39/1563 [..............................] - ETA: 2s - loss: 1.2406 - accuracy: 0.5673 77/1563 [>.............................] - ETA: 1s - loss: 1.2333 - accuracy: 0.5743 115/1563 [=>............................] - ETA: 1s - loss: 1.2473 - accuracy: 0.5663 153/1563 [=>............................] - ETA: 1s - loss: 1.2421 - accuracy: 0.5694 191/1563 [==>...........................] - ETA: 1s - loss: 1.2545 - accuracy: 0.5636 229/1563 [===>..........................] - ETA: 1s - loss: 1.2508 - accuracy: 0.5652 267/1563 [====>.........................] - ETA: 1s - loss: 1.2517 - accuracy: 0.5662 305/1563 [====>.........................] - ETA: 1s - loss: 1.2533 - accuracy: 0.5641 343/1563 [=====>........................] - ETA: 1s - loss: 1.2555 - accuracy: 0.5640 381/1563 [======>.......................] - ETA: 1s - loss: 1.2562 - accuracy: 0.5632 419/1563 [=======>......................] - ETA: 1s - loss: 1.2584 - accuracy: 0.5626 457/1563 [=======>......................] - ETA: 1s - loss: 1.2579 - accuracy: 0.5650 495/1563 [========>.....................] - ETA: 1s - loss: 1.2581 - accuracy: 0.5652 533/1563 [=========>....................] - ETA: 1s - loss: 1.2558 - accuracy: 0.5654 571/1563 [=========>....................] - ETA: 1s - loss: 1.2527 - accuracy: 0.5669 609/1563 [==========>...................] - ETA: 1s - loss: 1.2511 - accuracy: 0.5665 647/1563 [===========>..................] - ETA: 1s - loss: 1.2518 - accuracy: 0.5660 685/1563 [============>.................] - ETA: 1s - loss: 1.2544 - accuracy: 0.5635 723/1563 [============>.................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5635 761/1563 [=============>................] - ETA: 1s - loss: 1.2530 - accuracy: 0.5637 799/1563 [==============>...............] - ETA: 1s - loss: 1.2515 - accuracy: 0.5641 837/1563 [===============>..............] - ETA: 0s - loss: 1.2518 - accuracy: 0.5643 875/1563 [===============>..............] - ETA: 0s - loss: 1.2520 - accuracy: 0.5643 913/1563 [================>.............] - ETA: 0s - loss: 1.2524 - accuracy: 0.5640 951/1563 [=================>............] - ETA: 0s - loss: 1.2517 - accuracy: 0.5646 989/1563 [=================>............] - ETA: 0s - loss: 1.2520 - accuracy: 0.5642 1027/1563 [==================>...........] - ETA: 0s - loss: 1.2520 - accuracy: 0.5648 1065/1563 [===================>..........] - ETA: 0s - loss: 1.2514 - accuracy: 0.5646 1103/1563 [====================>.........] - ETA: 0s - loss: 1.2520 - accuracy: 0.5644 1141/1563 [====================>.........] - ETA: 0s - loss: 1.2524 - accuracy: 0.5645 1179/1563 [=====================>........] - ETA: 0s - loss: 1.2534 - accuracy: 0.5640 1217/1563 [======================>.......] - ETA: 0s - loss: 1.2516 - accuracy: 0.5646 1255/1563 [=======================>......] - ETA: 0s - loss: 1.2534 - accuracy: 0.5633 1293/1563 [=======================>......] - ETA: 0s - loss: 1.2530 - accuracy: 0.5633 1331/1563 [========================>.....] - ETA: 0s - loss: 1.2530 - accuracy: 0.5628 1369/1563 [=========================>....] - ETA: 0s - loss: 1.2534 - accuracy: 0.5625 1407/1563 [==========================>...] - ETA: 0s - loss: 1.2544 - accuracy: 0.5623 1445/1563 [==========================>...] - ETA: 0s - loss: 1.2537 - accuracy: 0.5623 1483/1563 [===========================>..] - ETA: 0s - loss: 1.2533 - accuracy: 0.5618 1521/1563 [============================>.] - ETA: 0s - loss: 1.2532 - accuracy: 0.5620 1558/1563 [============================>.] - ETA: 0s - loss: 1.2533 - accuracy: 0.5617 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2531 - accuracy: 0.5619 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 24.42150115966797 hydro03 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins8446579409734841768.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS