Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 96399 queued and waiting for resources srun: job 96399 has been allocated resources 2024-11-02 07:43:09.744431: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-02 07:43:43.989515: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-02 07:44:28.125919: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-02 07:44:28.126486: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-02 07:44:28.126495: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-02 07:45:37.332556: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-02 07:45:41.949517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-02 07:45:53.331527: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 2:29:23 - loss: 2.2975 - accuracy: 0.1875 38/1563 [..............................] - ETA: 2s - loss: 2.2360 - accuracy: 0.1637  77/1563 [>.............................] - ETA: 1s - loss: 2.1751 - accuracy: 0.1952 116/1563 [=>............................] - ETA: 1s - loss: 2.1343 - accuracy: 0.2171 156/1563 [=>............................] - ETA: 1s - loss: 2.0892 - accuracy: 0.2358 195/1563 [==>...........................] - ETA: 1s - loss: 2.0588 - accuracy: 0.2462 235/1563 [===>..........................] - ETA: 1s - loss: 2.0308 - accuracy: 0.2604 274/1563 [====>.........................] - ETA: 1s - loss: 2.0184 - accuracy: 0.2651 313/1563 [=====>........................] - ETA: 1s - loss: 2.0007 - accuracy: 0.2709 353/1563 [=====>........................] - ETA: 1s - loss: 1.9883 - accuracy: 0.2767 392/1563 [======>.......................] - ETA: 1s - loss: 1.9790 - accuracy: 0.2817 432/1563 [=======>......................] - ETA: 1s - loss: 1.9679 - accuracy: 0.2875 472/1563 [========>.....................] - ETA: 1s - loss: 1.9561 - accuracy: 0.2931 512/1563 [========>.....................] - ETA: 1s - loss: 1.9475 - accuracy: 0.2968 552/1563 [=========>....................] - ETA: 1s - loss: 1.9377 - accuracy: 0.3023 591/1563 [==========>...................] - ETA: 1s - loss: 1.9328 - accuracy: 0.3046 631/1563 [===========>..................] - ETA: 1s - loss: 1.9248 - accuracy: 0.3068 671/1563 [===========>..................] - ETA: 1s - loss: 1.9177 - accuracy: 0.3098 711/1563 [============>.................] - ETA: 1s - loss: 1.9113 - accuracy: 0.3125 751/1563 [=============>................] - ETA: 1s - loss: 1.9035 - accuracy: 0.3159 791/1563 [==============>...............] - ETA: 0s - loss: 1.8968 - accuracy: 0.3179 831/1563 [==============>...............] - ETA: 0s - loss: 1.8903 - accuracy: 0.3210 871/1563 [===============>..............] - ETA: 0s - loss: 1.8837 - accuracy: 0.3235 911/1563 [================>.............] - ETA: 0s - loss: 1.8775 - accuracy: 0.3258 950/1563 [=================>............] - ETA: 0s - loss: 1.8717 - accuracy: 0.3290 990/1563 [==================>...........] - ETA: 0s - loss: 1.8678 - accuracy: 0.3307 1029/1563 [==================>...........] - ETA: 0s - loss: 1.8643 - accuracy: 0.3321 1069/1563 [===================>..........] - ETA: 0s - loss: 1.8569 - accuracy: 0.3356 1109/1563 [====================>.........] - ETA: 0s - loss: 1.8517 - accuracy: 0.3388 1149/1563 [=====================>........] - ETA: 0s - loss: 1.8473 - accuracy: 0.3404 1188/1563 [=====================>........] - ETA: 0s - loss: 1.8446 - accuracy: 0.3419 1228/1563 [======================>.......] - ETA: 0s - loss: 1.8403 - accuracy: 0.3432 1267/1563 [=======================>......] - ETA: 0s - loss: 1.8370 - accuracy: 0.3450 1307/1563 [========================>.....] - ETA: 0s - loss: 1.8326 - accuracy: 0.3473 1347/1563 [========================>.....] - ETA: 0s - loss: 1.8306 - accuracy: 0.3480 1386/1563 [=========================>....] - ETA: 0s - loss: 1.8258 - accuracy: 0.3499 1426/1563 [==========================>...] - ETA: 0s - loss: 1.8228 - accuracy: 0.3514 1466/1563 [===========================>..] - ETA: 0s - loss: 1.8197 - accuracy: 0.3526 1505/1563 [===========================>..] - ETA: 0s - loss: 1.8163 - accuracy: 0.3540 1544/1563 [============================>.] - ETA: 0s - loss: 1.8136 - accuracy: 0.3547 1563/1563 [==============================] - 8s 1ms/step - loss: 1.8113 - accuracy: 0.3554 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.7535 - accuracy: 0.3438 41/1563 [..............................] - ETA: 1s - loss: 1.6638 - accuracy: 0.4230 80/1563 [>.............................] - ETA: 1s - loss: 1.6601 - accuracy: 0.4219 119/1563 [=>............................] - ETA: 1s - loss: 1.6556 - accuracy: 0.4212 159/1563 [==>...........................] - ETA: 1s - loss: 1.6560 - accuracy: 0.4153 199/1563 [==>...........................] - ETA: 1s - loss: 1.6598 - accuracy: 0.4154 239/1563 [===>..........................] - ETA: 1s - loss: 1.6538 - accuracy: 0.4192 279/1563 [====>.........................] - ETA: 1s - loss: 1.6533 - accuracy: 0.4133 319/1563 [=====>........................] - ETA: 1s - loss: 1.6521 - accuracy: 0.4142 358/1563 [=====>........................] - ETA: 1s - loss: 1.6495 - accuracy: 0.4145 397/1563 [======>.......................] - ETA: 1s - loss: 1.6478 - accuracy: 0.4142 436/1563 [=======>......................] - ETA: 1s - loss: 1.6481 - accuracy: 0.4154 475/1563 [========>.....................] - ETA: 1s - loss: 1.6472 - accuracy: 0.4174 514/1563 [========>.....................] - ETA: 1s - loss: 1.6443 - accuracy: 0.4175 553/1563 [=========>....................] - ETA: 1s - loss: 1.6409 - accuracy: 0.4179 594/1563 [==========>...................] - ETA: 1s - loss: 1.6405 - accuracy: 0.4170 634/1563 [===========>..................] - ETA: 1s - loss: 1.6369 - accuracy: 0.4184 674/1563 [===========>..................] - ETA: 1s - loss: 1.6376 - accuracy: 0.4184 714/1563 [============>.................] - ETA: 1s - loss: 1.6398 - accuracy: 0.4168 753/1563 [=============>................] - ETA: 1s - loss: 1.6409 - accuracy: 0.4172 793/1563 [==============>...............] - ETA: 0s - loss: 1.6392 - accuracy: 0.4188 833/1563 [==============>...............] - ETA: 0s - loss: 1.6391 - accuracy: 0.4184 873/1563 [===============>..............] - ETA: 0s - loss: 1.6377 - accuracy: 0.4197 913/1563 [================>.............] - ETA: 0s - loss: 1.6354 - accuracy: 0.4205 953/1563 [=================>............] - ETA: 0s - loss: 1.6336 - accuracy: 0.4207 992/1563 [==================>...........] - ETA: 0s - loss: 1.6340 - accuracy: 0.4209 1031/1563 [==================>...........] - ETA: 0s - loss: 1.6341 - accuracy: 0.4211 1070/1563 [===================>..........] - ETA: 0s - loss: 1.6325 - accuracy: 0.4217 1109/1563 [====================>.........] - ETA: 0s - loss: 1.6311 - accuracy: 0.4226 1149/1563 [=====================>........] - ETA: 0s - loss: 1.6300 - accuracy: 0.4232 1189/1563 [=====================>........] - ETA: 0s - loss: 1.6292 - accuracy: 0.4235 1229/1563 [======================>.......] - ETA: 0s - loss: 1.6289 - accuracy: 0.4241 1269/1563 [=======================>......] - ETA: 0s - loss: 1.6272 - accuracy: 0.4248 1309/1563 [========================>.....] - ETA: 0s - loss: 1.6262 - accuracy: 0.4256 1349/1563 [========================>.....] - ETA: 0s - loss: 1.6261 - accuracy: 0.4258 1388/1563 [=========================>....] - ETA: 0s - loss: 1.6248 - accuracy: 0.4264 1427/1563 [==========================>...] - ETA: 0s - loss: 1.6237 - accuracy: 0.4271 1467/1563 [===========================>..] - ETA: 0s - loss: 1.6233 - accuracy: 0.4275 1507/1563 [===========================>..] - ETA: 0s - loss: 1.6217 - accuracy: 0.4284 1547/1563 [============================>.] - ETA: 0s - loss: 1.6212 - accuracy: 0.4280 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6210 - accuracy: 0.4279 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.5679 - accuracy: 0.3750 41/1563 [..............................] - ETA: 1s - loss: 1.5934 - accuracy: 0.4398 81/1563 [>.............................] - ETA: 1s - loss: 1.5497 - accuracy: 0.4564 121/1563 [=>............................] - ETA: 1s - loss: 1.5518 - accuracy: 0.4566 161/1563 [==>...........................] - ETA: 1s - loss: 1.5434 - accuracy: 0.4573 200/1563 [==>...........................] - ETA: 1s - loss: 1.5464 - accuracy: 0.4548 240/1563 [===>..........................] - ETA: 1s - loss: 1.5465 - accuracy: 0.4569 279/1563 [====>.........................] - ETA: 1s - loss: 1.5435 - accuracy: 0.4603 318/1563 [=====>........................] - ETA: 1s - loss: 1.5471 - accuracy: 0.4590 357/1563 [=====>........................] - ETA: 1s - loss: 1.5495 - accuracy: 0.4550 396/1563 [======>.......................] - ETA: 1s - loss: 1.5477 - accuracy: 0.4558 436/1563 [=======>......................] - ETA: 1s - loss: 1.5476 - accuracy: 0.4545 476/1563 [========>.....................] - ETA: 1s - loss: 1.5465 - accuracy: 0.4555 516/1563 [========>.....................] - ETA: 1s - loss: 1.5469 - accuracy: 0.4558 556/1563 [=========>....................] - ETA: 1s - loss: 1.5454 - accuracy: 0.4566 596/1563 [==========>...................] - ETA: 1s - loss: 1.5455 - accuracy: 0.4570 636/1563 [===========>..................] - ETA: 1s - loss: 1.5444 - accuracy: 0.4577 676/1563 [===========>..................] - ETA: 1s - loss: 1.5467 - accuracy: 0.4566 716/1563 [============>.................] - ETA: 1s - loss: 1.5460 - accuracy: 0.4568 755/1563 [=============>................] - ETA: 1s - loss: 1.5454 - accuracy: 0.4568 795/1563 [==============>...............] - ETA: 0s - loss: 1.5457 - accuracy: 0.4565 835/1563 [===============>..............] - ETA: 0s - loss: 1.5455 - accuracy: 0.4554 875/1563 [===============>..............] - ETA: 0s - loss: 1.5446 - accuracy: 0.4552 914/1563 [================>.............] - ETA: 0s - loss: 1.5453 - accuracy: 0.4545 954/1563 [=================>............] - ETA: 0s - loss: 1.5452 - accuracy: 0.4546 994/1563 [==================>...........] - ETA: 0s - loss: 1.5448 - accuracy: 0.4544 1034/1563 [==================>...........] - ETA: 0s - loss: 1.5441 - accuracy: 0.4543 1073/1563 [===================>..........] - ETA: 0s - loss: 1.5448 - accuracy: 0.4544 1113/1563 [====================>.........] - ETA: 0s - loss: 1.5444 - accuracy: 0.4551 1153/1563 [=====================>........] - ETA: 0s - loss: 1.5435 - accuracy: 0.4557 1193/1563 [=====================>........] - ETA: 0s - loss: 1.5421 - accuracy: 0.4565 1233/1563 [======================>.......] - ETA: 0s - loss: 1.5421 - accuracy: 0.4564 1271/1563 [=======================>......] - ETA: 0s - loss: 1.5414 - accuracy: 0.4564 1308/1563 [========================>.....] - ETA: 0s - loss: 1.5428 - accuracy: 0.4560 1344/1563 [========================>.....] - ETA: 0s - loss: 1.5415 - accuracy: 0.4566 1381/1563 [=========================>....] - ETA: 0s - loss: 1.5410 - accuracy: 0.4566 1418/1563 [==========================>...] - ETA: 0s - loss: 1.5413 - accuracy: 0.4570 1454/1563 [==========================>...] - ETA: 0s - loss: 1.5407 - accuracy: 0.4570 1491/1563 [===========================>..] - ETA: 0s - loss: 1.5412 - accuracy: 0.4570 1528/1563 [============================>.] - ETA: 0s - loss: 1.5402 - accuracy: 0.4572 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5402 - accuracy: 0.4573 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.3444 - accuracy: 0.4688 37/1563 [..............................] - ETA: 2s - loss: 1.4862 - accuracy: 0.4704 74/1563 [>.............................] - ETA: 2s - loss: 1.4841 - accuracy: 0.4759 109/1563 [=>............................] - ETA: 2s - loss: 1.4967 - accuracy: 0.4653 145/1563 [=>............................] - ETA: 1s - loss: 1.4923 - accuracy: 0.4668 182/1563 [==>...........................] - ETA: 1s - loss: 1.4930 - accuracy: 0.4700 218/1563 [===>..........................] - ETA: 1s - loss: 1.4920 - accuracy: 0.4700 254/1563 [===>..........................] - ETA: 1s - loss: 1.4949 - accuracy: 0.4678 290/1563 [====>.........................] - ETA: 1s - loss: 1.4923 - accuracy: 0.4686 326/1563 [=====>........................] - ETA: 1s - loss: 1.4893 - accuracy: 0.4691 362/1563 [=====>........................] - ETA: 1s - loss: 1.4855 - accuracy: 0.4701 399/1563 [======>.......................] - ETA: 1s - loss: 1.4828 - accuracy: 0.4716 436/1563 [=======>......................] - ETA: 1s - loss: 1.4838 - accuracy: 0.4735 473/1563 [========>.....................] - ETA: 1s - loss: 1.4846 - accuracy: 0.4731 510/1563 [========>.....................] - ETA: 1s - loss: 1.4798 - accuracy: 0.4754 546/1563 [=========>....................] - ETA: 1s - loss: 1.4798 - accuracy: 0.4747 582/1563 [==========>...................] - ETA: 1s - loss: 1.4777 - accuracy: 0.4765 618/1563 [==========>...................] - ETA: 1s - loss: 1.4759 - accuracy: 0.4776 655/1563 [===========>..................] - ETA: 1s - loss: 1.4778 - accuracy: 0.4765 692/1563 [============>.................] - ETA: 1s - loss: 1.4761 - accuracy: 0.4776 730/1563 [=============>................] - ETA: 1s - loss: 1.4760 - accuracy: 0.4773 767/1563 [=============>................] - ETA: 1s - loss: 1.4755 - accuracy: 0.4776 803/1563 [==============>...............] - ETA: 1s - loss: 1.4772 - accuracy: 0.4772 840/1563 [===============>..............] - ETA: 1s - loss: 1.4767 - accuracy: 0.4784 877/1563 [===============>..............] - ETA: 0s - loss: 1.4755 - accuracy: 0.4788 914/1563 [================>.............] - ETA: 0s - loss: 1.4763 - accuracy: 0.4782 951/1563 [=================>............] - ETA: 0s - loss: 1.4765 - accuracy: 0.4785 988/1563 [=================>............] - ETA: 0s - loss: 1.4774 - accuracy: 0.4788 1024/1563 [==================>...........] - ETA: 0s - loss: 1.4783 - accuracy: 0.4784 1061/1563 [===================>..........] - ETA: 0s - loss: 1.4794 - accuracy: 0.4779 1100/1563 [====================>.........] - ETA: 0s - loss: 1.4792 - accuracy: 0.4782 1137/1563 [====================>.........] - ETA: 0s - loss: 1.4795 - accuracy: 0.4785 1174/1563 [=====================>........] - ETA: 0s - loss: 1.4785 - accuracy: 0.4785 1211/1563 [======================>.......] - ETA: 0s - loss: 1.4782 - accuracy: 0.4791 1248/1563 [======================>.......] - ETA: 0s - loss: 1.4779 - accuracy: 0.4793 1285/1563 [=======================>......] - ETA: 0s - loss: 1.4774 - accuracy: 0.4797 1323/1563 [========================>.....] - ETA: 0s - loss: 1.4777 - accuracy: 0.4796 1361/1563 [=========================>....] - ETA: 0s - loss: 1.4771 - accuracy: 0.4804 1399/1563 [=========================>....] - ETA: 0s - loss: 1.4776 - accuracy: 0.4797 1437/1563 [==========================>...] - ETA: 0s - loss: 1.4780 - accuracy: 0.4795 1475/1563 [===========================>..] - ETA: 0s - loss: 1.4778 - accuracy: 0.4799 1512/1563 [============================>.] - ETA: 0s - loss: 1.4796 - accuracy: 0.4795 1549/1563 [============================>.] - ETA: 0s - loss: 1.4792 - accuracy: 0.4794 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4787 - accuracy: 0.4795 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.6277 - accuracy: 0.3438 38/1563 [..............................] - ETA: 2s - loss: 1.4271 - accuracy: 0.4934 75/1563 [>.............................] - ETA: 2s - loss: 1.4513 - accuracy: 0.4896 112/1563 [=>............................] - ETA: 1s - loss: 1.4411 - accuracy: 0.4939 149/1563 [=>............................] - ETA: 1s - loss: 1.4426 - accuracy: 0.4908 186/1563 [==>...........................] - ETA: 1s - loss: 1.4351 - accuracy: 0.4955 223/1563 [===>..........................] - ETA: 1s - loss: 1.4386 - accuracy: 0.4971 260/1563 [===>..........................] - ETA: 1s - loss: 1.4333 - accuracy: 0.4998 297/1563 [====>.........................] - ETA: 1s - loss: 1.4314 - accuracy: 0.4987 335/1563 [=====>........................] - ETA: 1s - loss: 1.4391 - accuracy: 0.4956 373/1563 [======>.......................] - ETA: 1s - loss: 1.4347 - accuracy: 0.4974 411/1563 [======>.......................] - ETA: 1s - loss: 1.4321 - accuracy: 0.4987 450/1563 [=======>......................] - ETA: 1s - loss: 1.4344 - accuracy: 0.4976 488/1563 [========>.....................] - ETA: 1s - loss: 1.4337 - accuracy: 0.4969 526/1563 [=========>....................] - ETA: 1s - loss: 1.4370 - accuracy: 0.4959 563/1563 [=========>....................] - ETA: 1s - loss: 1.4393 - accuracy: 0.4941 600/1563 [==========>...................] - ETA: 1s - loss: 1.4405 - accuracy: 0.4939 636/1563 [===========>..................] - ETA: 1s - loss: 1.4389 - accuracy: 0.4942 672/1563 [===========>..................] - ETA: 1s - loss: 1.4391 - accuracy: 0.4944 709/1563 [============>.................] - ETA: 1s - loss: 1.4414 - accuracy: 0.4937 746/1563 [=============>................] - ETA: 1s - loss: 1.4393 - accuracy: 0.4946 784/1563 [==============>...............] - ETA: 1s - loss: 1.4410 - accuracy: 0.4939 820/1563 [==============>...............] - ETA: 1s - loss: 1.4404 - accuracy: 0.4938 858/1563 [===============>..............] - ETA: 0s - loss: 1.4396 - accuracy: 0.4944 896/1563 [================>.............] - ETA: 0s - loss: 1.4397 - accuracy: 0.4943 934/1563 [================>.............] - ETA: 0s - loss: 1.4382 - accuracy: 0.4958 971/1563 [=================>............] - ETA: 0s - loss: 1.4375 - accuracy: 0.4959 1009/1563 [==================>...........] - ETA: 0s - loss: 1.4381 - accuracy: 0.4950 1048/1563 [===================>..........] - ETA: 0s - loss: 1.4380 - accuracy: 0.4955 1085/1563 [===================>..........] - ETA: 0s - loss: 1.4351 - accuracy: 0.4967 1124/1563 [====================>.........] - ETA: 0s - loss: 1.4369 - accuracy: 0.4961 1162/1563 [=====================>........] - ETA: 0s - loss: 1.4347 - accuracy: 0.4964 1200/1563 [======================>.......] - ETA: 0s - loss: 1.4339 - accuracy: 0.4966 1239/1563 [======================>.......] - ETA: 0s - loss: 1.4327 - accuracy: 0.4967 1276/1563 [=======================>......] - ETA: 0s - loss: 1.4329 - accuracy: 0.4968 1314/1563 [========================>.....] - ETA: 0s - loss: 1.4330 - accuracy: 0.4969 1351/1563 [========================>.....] - ETA: 0s - loss: 1.4335 - accuracy: 0.4970 1389/1563 [=========================>....] - ETA: 0s - loss: 1.4331 - accuracy: 0.4971 1427/1563 [==========================>...] - ETA: 0s - loss: 1.4323 - accuracy: 0.4969 1464/1563 [===========================>..] - ETA: 0s - loss: 1.4317 - accuracy: 0.4972 1502/1563 [===========================>..] - ETA: 0s - loss: 1.4315 - accuracy: 0.4972 1540/1563 [============================>.] - ETA: 0s - loss: 1.4315 - accuracy: 0.4976 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4315 - accuracy: 0.4974 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.2055 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.3310 - accuracy: 0.5264 78/1563 [>.............................] - ETA: 1s - loss: 1.3633 - accuracy: 0.5144 115/1563 [=>............................] - ETA: 1s - loss: 1.3664 - accuracy: 0.5101 153/1563 [=>............................] - ETA: 1s - loss: 1.3881 - accuracy: 0.5076 191/1563 [==>...........................] - ETA: 1s - loss: 1.3900 - accuracy: 0.5075 229/1563 [===>..........................] - ETA: 1s - loss: 1.3930 - accuracy: 0.5085 266/1563 [====>.........................] - ETA: 1s - loss: 1.3936 - accuracy: 0.5070 303/1563 [====>.........................] - ETA: 1s - loss: 1.3904 - accuracy: 0.5101 341/1563 [=====>........................] - ETA: 1s - loss: 1.3944 - accuracy: 0.5101 379/1563 [======>.......................] - ETA: 1s - loss: 1.4011 - accuracy: 0.5068 417/1563 [=======>......................] - ETA: 1s - loss: 1.4007 - accuracy: 0.5070 456/1563 [=======>......................] - ETA: 1s - loss: 1.3993 - accuracy: 0.5065 494/1563 [========>.....................] - ETA: 1s - loss: 1.4008 - accuracy: 0.5068 531/1563 [=========>....................] - ETA: 1s - loss: 1.3977 - accuracy: 0.5081 568/1563 [=========>....................] - ETA: 1s - loss: 1.3975 - accuracy: 0.5092 606/1563 [==========>...................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5097 643/1563 [===========>..................] - ETA: 1s - loss: 1.3930 - accuracy: 0.5107 680/1563 [============>.................] - ETA: 1s - loss: 1.3891 - accuracy: 0.5118 718/1563 [============>.................] - ETA: 1s - loss: 1.3901 - accuracy: 0.5120 755/1563 [=============>................] - ETA: 1s - loss: 1.3892 - accuracy: 0.5121 793/1563 [==============>...............] - ETA: 1s - loss: 1.3903 - accuracy: 0.5116 831/1563 [==============>...............] - ETA: 0s - loss: 1.3888 - accuracy: 0.5121 869/1563 [===============>..............] - ETA: 0s - loss: 1.3886 - accuracy: 0.5123 907/1563 [================>.............] - ETA: 0s - loss: 1.3869 - accuracy: 0.5124 945/1563 [=================>............] - ETA: 0s - loss: 1.3881 - accuracy: 0.5120 983/1563 [=================>............] - ETA: 0s - loss: 1.3883 - accuracy: 0.5120 1020/1563 [==================>...........] - ETA: 0s - loss: 1.3861 - accuracy: 0.5130 1057/1563 [===================>..........] - ETA: 0s - loss: 1.3854 - accuracy: 0.5131 1094/1563 [===================>..........] - ETA: 0s - loss: 1.3836 - accuracy: 0.5136 1131/1563 [====================>.........] - ETA: 0s - loss: 1.3835 - accuracy: 0.5132 1169/1563 [=====================>........] - ETA: 0s - loss: 1.3824 - accuracy: 0.5133 1206/1563 [======================>.......] - ETA: 0s - loss: 1.3842 - accuracy: 0.5132 1244/1563 [======================>.......] - ETA: 0s - loss: 1.3847 - accuracy: 0.5130 1281/1563 [=======================>......] - ETA: 0s - loss: 1.3843 - accuracy: 0.5129 1319/1563 [========================>.....] - ETA: 0s - loss: 1.3854 - accuracy: 0.5129 1356/1563 [=========================>....] - ETA: 0s - loss: 1.3850 - accuracy: 0.5134 1394/1563 [=========================>....] - ETA: 0s - loss: 1.3849 - accuracy: 0.5134 1433/1563 [==========================>...] - ETA: 0s - loss: 1.3855 - accuracy: 0.5134 1471/1563 [===========================>..] - ETA: 0s - loss: 1.3855 - accuracy: 0.5133 1509/1563 [===========================>..] - ETA: 0s - loss: 1.3866 - accuracy: 0.5127 1546/1563 [============================>.] - ETA: 0s - loss: 1.3862 - accuracy: 0.5126 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3865 - accuracy: 0.5126 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.5660 - accuracy: 0.5000 39/1563 [..............................] - ETA: 2s - loss: 1.3807 - accuracy: 0.5136 77/1563 [>.............................] - ETA: 1s - loss: 1.3721 - accuracy: 0.5191 114/1563 [=>............................] - ETA: 1s - loss: 1.3622 - accuracy: 0.5186 152/1563 [=>............................] - ETA: 1s - loss: 1.3641 - accuracy: 0.5158 190/1563 [==>...........................] - ETA: 1s - loss: 1.3564 - accuracy: 0.5197 228/1563 [===>..........................] - ETA: 1s - loss: 1.3627 - accuracy: 0.5215 264/1563 [====>.........................] - ETA: 1s - loss: 1.3585 - accuracy: 0.5245 301/1563 [====>.........................] - ETA: 1s - loss: 1.3562 - accuracy: 0.5268 337/1563 [=====>........................] - ETA: 1s - loss: 1.3583 - accuracy: 0.5251 375/1563 [======>.......................] - ETA: 1s - loss: 1.3587 - accuracy: 0.5238 414/1563 [======>.......................] - ETA: 1s - loss: 1.3588 - accuracy: 0.5245 452/1563 [=======>......................] - ETA: 1s - loss: 1.3641 - accuracy: 0.5224 490/1563 [========>.....................] - ETA: 1s - loss: 1.3607 - accuracy: 0.5242 528/1563 [=========>....................] - ETA: 1s - loss: 1.3567 - accuracy: 0.5257 566/1563 [=========>....................] - ETA: 1s - loss: 1.3566 - accuracy: 0.5264 604/1563 [==========>...................] - ETA: 1s - loss: 1.3558 - accuracy: 0.5275 641/1563 [===========>..................] - ETA: 1s - loss: 1.3543 - accuracy: 0.5287 678/1563 [============>.................] - ETA: 1s - loss: 1.3535 - accuracy: 0.5284 716/1563 [============>.................] - ETA: 1s - loss: 1.3543 - accuracy: 0.5276 754/1563 [=============>................] - ETA: 1s - loss: 1.3536 - accuracy: 0.5274 792/1563 [==============>...............] - ETA: 1s - loss: 1.3538 - accuracy: 0.5266 831/1563 [==============>...............] - ETA: 0s - loss: 1.3532 - accuracy: 0.5260 869/1563 [===============>..............] - ETA: 0s - loss: 1.3522 - accuracy: 0.5266 907/1563 [================>.............] - ETA: 0s - loss: 1.3518 - accuracy: 0.5269 944/1563 [=================>............] - ETA: 0s - loss: 1.3496 - accuracy: 0.5274 981/1563 [=================>............] - ETA: 0s - loss: 1.3492 - accuracy: 0.5271 1019/1563 [==================>...........] - ETA: 0s - loss: 1.3480 - accuracy: 0.5276 1056/1563 [===================>..........] - ETA: 0s - loss: 1.3492 - accuracy: 0.5273 1093/1563 [===================>..........] - ETA: 0s - loss: 1.3476 - accuracy: 0.5275 1130/1563 [====================>.........] - ETA: 0s - loss: 1.3475 - accuracy: 0.5277 1167/1563 [=====================>........] - ETA: 0s - loss: 1.3483 - accuracy: 0.5280 1203/1563 [======================>.......] - ETA: 0s - loss: 1.3476 - accuracy: 0.5281 1240/1563 [======================>.......] - ETA: 0s - loss: 1.3475 - accuracy: 0.5284 1277/1563 [=======================>......] - ETA: 0s - loss: 1.3473 - accuracy: 0.5284 1315/1563 [========================>.....] - ETA: 0s - loss: 1.3477 - accuracy: 0.5281 1353/1563 [========================>.....] - ETA: 0s - loss: 1.3479 - accuracy: 0.5279 1391/1563 [=========================>....] - ETA: 0s - loss: 1.3472 - accuracy: 0.5282 1428/1563 [==========================>...] - ETA: 0s - loss: 1.3477 - accuracy: 0.5280 1465/1563 [===========================>..] - ETA: 0s - loss: 1.3485 - accuracy: 0.5279 1503/1563 [===========================>..] - ETA: 0s - loss: 1.3491 - accuracy: 0.5271 1541/1563 [============================>.] - ETA: 0s - loss: 1.3492 - accuracy: 0.5271 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3493 - accuracy: 0.5272 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.1225 - accuracy: 0.6562 39/1563 [..............................] - ETA: 2s - loss: 1.3086 - accuracy: 0.5425 76/1563 [>.............................] - ETA: 2s - loss: 1.3134 - accuracy: 0.5473 114/1563 [=>............................] - ETA: 1s - loss: 1.3096 - accuracy: 0.5428 152/1563 [=>............................] - ETA: 1s - loss: 1.3079 - accuracy: 0.5438 189/1563 [==>...........................] - ETA: 1s - loss: 1.3108 - accuracy: 0.5451 227/1563 [===>..........................] - ETA: 1s - loss: 1.3128 - accuracy: 0.5443 265/1563 [====>.........................] - ETA: 1s - loss: 1.3083 - accuracy: 0.5447 302/1563 [====>.........................] - ETA: 1s - loss: 1.3107 - accuracy: 0.5430 339/1563 [=====>........................] - ETA: 1s - loss: 1.3128 - accuracy: 0.5428 376/1563 [======>.......................] - ETA: 1s - loss: 1.3199 - accuracy: 0.5383 414/1563 [======>.......................] - ETA: 1s - loss: 1.3193 - accuracy: 0.5394 452/1563 [=======>......................] - ETA: 1s - loss: 1.3184 - accuracy: 0.5400 490/1563 [========>.....................] - ETA: 1s - loss: 1.3155 - accuracy: 0.5405 527/1563 [=========>....................] - ETA: 1s - loss: 1.3150 - accuracy: 0.5406 564/1563 [=========>....................] - ETA: 1s - loss: 1.3140 - accuracy: 0.5393 600/1563 [==========>...................] - ETA: 1s - loss: 1.3152 - accuracy: 0.5390 637/1563 [===========>..................] - ETA: 1s - loss: 1.3178 - accuracy: 0.5383 675/1563 [===========>..................] - ETA: 1s - loss: 1.3202 - accuracy: 0.5374 712/1563 [============>.................] - ETA: 1s - loss: 1.3194 - accuracy: 0.5374 748/1563 [=============>................] - ETA: 1s - loss: 1.3174 - accuracy: 0.5372 785/1563 [==============>...............] - ETA: 1s - loss: 1.3185 - accuracy: 0.5365 822/1563 [==============>...............] - ETA: 1s - loss: 1.3193 - accuracy: 0.5366 860/1563 [===============>..............] - ETA: 0s - loss: 1.3205 - accuracy: 0.5355 898/1563 [================>.............] - ETA: 0s - loss: 1.3190 - accuracy: 0.5356 935/1563 [================>.............] - ETA: 0s - loss: 1.3200 - accuracy: 0.5355 972/1563 [=================>............] - ETA: 0s - loss: 1.3210 - accuracy: 0.5351 1009/1563 [==================>...........] - ETA: 0s - loss: 1.3213 - accuracy: 0.5347 1046/1563 [===================>..........] - ETA: 0s - loss: 1.3199 - accuracy: 0.5350 1084/1563 [===================>..........] - ETA: 0s - loss: 1.3188 - accuracy: 0.5352 1122/1563 [====================>.........] - ETA: 0s - loss: 1.3193 - accuracy: 0.5352 1159/1563 [=====================>........] - ETA: 0s - loss: 1.3179 - accuracy: 0.5362 1196/1563 [=====================>........] - ETA: 0s - loss: 1.3182 - accuracy: 0.5362 1234/1563 [======================>.......] - ETA: 0s - loss: 1.3165 - accuracy: 0.5371 1272/1563 [=======================>......] - ETA: 0s - loss: 1.3170 - accuracy: 0.5371 1310/1563 [========================>.....] - ETA: 0s - loss: 1.3182 - accuracy: 0.5364 1348/1563 [========================>.....] - ETA: 0s - loss: 1.3180 - accuracy: 0.5368 1386/1563 [=========================>....] - ETA: 0s - loss: 1.3164 - accuracy: 0.5375 1424/1563 [==========================>...] - ETA: 0s - loss: 1.3167 - accuracy: 0.5372 1461/1563 [===========================>..] - ETA: 0s - loss: 1.3161 - accuracy: 0.5376 1498/1563 [===========================>..] - ETA: 0s - loss: 1.3163 - accuracy: 0.5376 1535/1563 [============================>.] - ETA: 0s - loss: 1.3157 - accuracy: 0.5380 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3150 - accuracy: 0.5386 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.3919 - accuracy: 0.5312 38/1563 [..............................] - ETA: 2s - loss: 1.2753 - accuracy: 0.5452 75/1563 [>.............................] - ETA: 2s - loss: 1.2682 - accuracy: 0.5638 113/1563 [=>............................] - ETA: 1s - loss: 1.2590 - accuracy: 0.5611 151/1563 [=>............................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5627 187/1563 [==>...........................] - ETA: 1s - loss: 1.2623 - accuracy: 0.5603 225/1563 [===>..........................] - ETA: 1s - loss: 1.2695 - accuracy: 0.5579 262/1563 [====>.........................] - ETA: 1s - loss: 1.2756 - accuracy: 0.5565 299/1563 [====>.........................] - ETA: 1s - loss: 1.2755 - accuracy: 0.5551 335/1563 [=====>........................] - ETA: 1s - loss: 1.2793 - accuracy: 0.5552 371/1563 [======>.......................] - ETA: 1s - loss: 1.2784 - accuracy: 0.5548 408/1563 [======>.......................] - ETA: 1s - loss: 1.2760 - accuracy: 0.5547 445/1563 [=======>......................] - ETA: 1s - loss: 1.2735 - accuracy: 0.5551 481/1563 [========>.....................] - ETA: 1s - loss: 1.2750 - accuracy: 0.5535 519/1563 [========>.....................] - ETA: 1s - loss: 1.2728 - accuracy: 0.5535 555/1563 [=========>....................] - ETA: 1s - loss: 1.2759 - accuracy: 0.5524 591/1563 [==========>...................] - ETA: 1s - loss: 1.2731 - accuracy: 0.5535 628/1563 [===========>..................] - ETA: 1s - loss: 1.2741 - accuracy: 0.5529 664/1563 [===========>..................] - ETA: 1s - loss: 1.2746 - accuracy: 0.5519 701/1563 [============>.................] - ETA: 1s - loss: 1.2763 - accuracy: 0.5518 739/1563 [=============>................] - ETA: 1s - loss: 1.2776 - accuracy: 0.5526 775/1563 [=============>................] - ETA: 1s - loss: 1.2784 - accuracy: 0.5515 813/1563 [==============>...............] - ETA: 1s - loss: 1.2776 - accuracy: 0.5510 851/1563 [===============>..............] - ETA: 0s - loss: 1.2776 - accuracy: 0.5505 889/1563 [================>.............] - ETA: 0s - loss: 1.2783 - accuracy: 0.5502 926/1563 [================>.............] - ETA: 0s - loss: 1.2771 - accuracy: 0.5504 963/1563 [=================>............] - ETA: 0s - loss: 1.2790 - accuracy: 0.5507 1001/1563 [==================>...........] - ETA: 0s - loss: 1.2799 - accuracy: 0.5500 1039/1563 [==================>...........] - ETA: 0s - loss: 1.2782 - accuracy: 0.5507 1077/1563 [===================>..........] - ETA: 0s - loss: 1.2766 - accuracy: 0.5515 1114/1563 [====================>.........] - ETA: 0s - loss: 1.2794 - accuracy: 0.5504 1152/1563 [=====================>........] - ETA: 0s - loss: 1.2784 - accuracy: 0.5509 1189/1563 [=====================>........] - ETA: 0s - loss: 1.2804 - accuracy: 0.5500 1226/1563 [======================>.......] - ETA: 0s - loss: 1.2811 - accuracy: 0.5495 1264/1563 [=======================>......] - ETA: 0s - loss: 1.2808 - accuracy: 0.5497 1301/1563 [=======================>......] - ETA: 0s - loss: 1.2816 - accuracy: 0.5496 1338/1563 [========================>.....] - ETA: 0s - loss: 1.2840 - accuracy: 0.5487 1376/1563 [=========================>....] - ETA: 0s - loss: 1.2839 - accuracy: 0.5484 1413/1563 [==========================>...] - ETA: 0s - loss: 1.2831 - accuracy: 0.5484 1449/1563 [==========================>...] - ETA: 0s - loss: 1.2830 - accuracy: 0.5487 1486/1563 [===========================>..] - ETA: 0s - loss: 1.2824 - accuracy: 0.5489 1523/1563 [============================>.] - ETA: 0s - loss: 1.2824 - accuracy: 0.5493 1560/1563 [============================>.] - ETA: 0s - loss: 1.2825 - accuracy: 0.5496 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2826 - accuracy: 0.5495 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.0602 - accuracy: 0.6562 38/1563 [..............................] - ETA: 2s - loss: 1.2293 - accuracy: 0.5666 74/1563 [>.............................] - ETA: 2s - loss: 1.2248 - accuracy: 0.5676 110/1563 [=>............................] - ETA: 2s - loss: 1.2401 - accuracy: 0.5679 147/1563 [=>............................] - ETA: 1s - loss: 1.2404 - accuracy: 0.5644 183/1563 [==>...........................] - ETA: 1s - loss: 1.2394 - accuracy: 0.5656 220/1563 [===>..........................] - ETA: 1s - loss: 1.2350 - accuracy: 0.5678 256/1563 [===>..........................] - ETA: 1s - loss: 1.2331 - accuracy: 0.5691 293/1563 [====>.........................] - ETA: 1s - loss: 1.2427 - accuracy: 0.5637 329/1563 [=====>........................] - ETA: 1s - loss: 1.2403 - accuracy: 0.5644 365/1563 [======>.......................] - ETA: 1s - loss: 1.2382 - accuracy: 0.5662 402/1563 [======>.......................] - ETA: 1s - loss: 1.2375 - accuracy: 0.5646 439/1563 [=======>......................] - ETA: 1s - loss: 1.2374 - accuracy: 0.5652 475/1563 [========>.....................] - ETA: 1s - loss: 1.2446 - accuracy: 0.5636 511/1563 [========>.....................] - ETA: 1s - loss: 1.2451 - accuracy: 0.5626 548/1563 [=========>....................] - ETA: 1s - loss: 1.2469 - accuracy: 0.5624 584/1563 [==========>...................] - ETA: 1s - loss: 1.2491 - accuracy: 0.5609 621/1563 [==========>...................] - ETA: 1s - loss: 1.2491 - accuracy: 0.5607 657/1563 [===========>..................] - ETA: 1s - loss: 1.2481 - accuracy: 0.5611 694/1563 [============>.................] - ETA: 1s - loss: 1.2481 - accuracy: 0.5614 730/1563 [=============>................] - ETA: 1s - loss: 1.2472 - accuracy: 0.5616 767/1563 [=============>................] - ETA: 1s - loss: 1.2503 - accuracy: 0.5599 804/1563 [==============>...............] - ETA: 1s - loss: 1.2494 - accuracy: 0.5599 841/1563 [===============>..............] - ETA: 1s - loss: 1.2500 - accuracy: 0.5598 878/1563 [===============>..............] - ETA: 0s - loss: 1.2502 - accuracy: 0.5601 915/1563 [================>.............] - ETA: 0s - loss: 1.2514 - accuracy: 0.5598 952/1563 [=================>............] - ETA: 0s - loss: 1.2502 - accuracy: 0.5600 988/1563 [=================>............] - ETA: 0s - loss: 1.2500 - accuracy: 0.5600 1024/1563 [==================>...........] - ETA: 0s - loss: 1.2480 - accuracy: 0.5608 1061/1563 [===================>..........] - ETA: 0s - loss: 1.2475 - accuracy: 0.5610 1098/1563 [====================>.........] - ETA: 0s - loss: 1.2469 - accuracy: 0.5607 1135/1563 [====================>.........] - ETA: 0s - loss: 1.2482 - accuracy: 0.5602 1171/1563 [=====================>........] - ETA: 0s - loss: 1.2484 - accuracy: 0.5598 1207/1563 [======================>.......] - ETA: 0s - loss: 1.2491 - accuracy: 0.5600 1244/1563 [======================>.......] - ETA: 0s - loss: 1.2474 - accuracy: 0.5608 1280/1563 [=======================>......] - ETA: 0s - loss: 1.2467 - accuracy: 0.5613 1317/1563 [========================>.....] - ETA: 0s - loss: 1.2472 - accuracy: 0.5610 1354/1563 [========================>.....] - ETA: 0s - loss: 1.2483 - accuracy: 0.5607 1391/1563 [=========================>....] - ETA: 0s - loss: 1.2484 - accuracy: 0.5611 1428/1563 [==========================>...] - ETA: 0s - loss: 1.2486 - accuracy: 0.5615 1465/1563 [===========================>..] - ETA: 0s - loss: 1.2489 - accuracy: 0.5612 1502/1563 [===========================>..] - ETA: 0s - loss: 1.2487 - accuracy: 0.5618 1539/1563 [============================>.] - ETA: 0s - loss: 1.2490 - accuracy: 0.5617 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2492 - accuracy: 0.5616 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 44.35474491119385 hydro03 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins11259939867803497991.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS