Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 96453 queued and waiting for resources srun: job 96453 has been allocated resources 2024-11-03 07:43:09.551988: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-03 07:43:51.661861: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-03 07:44:40.289945: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-03 07:44:40.290801: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-03 07:44:40.290811: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-03 07:45:59.324882: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-03 07:46:03.982839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-03 07:46:15.887697: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 2:28:28 - loss: 2.3118 - accuracy: 0.0938 37/1563 [..............................] - ETA: 2s - loss: 2.2478 - accuracy: 0.1630  74/1563 [>.............................] - ETA: 2s - loss: 2.1828 - accuracy: 0.1913 112/1563 [=>............................] - ETA: 1s - loss: 2.1315 - accuracy: 0.2137 150/1563 [=>............................] - ETA: 1s - loss: 2.0962 - accuracy: 0.2265 188/1563 [==>...........................] - ETA: 1s - loss: 2.0733 - accuracy: 0.2357 226/1563 [===>..........................] - ETA: 1s - loss: 2.0494 - accuracy: 0.2474 264/1563 [====>.........................] - ETA: 1s - loss: 2.0287 - accuracy: 0.2582 303/1563 [====>.........................] - ETA: 1s - loss: 2.0156 - accuracy: 0.2638 342/1563 [=====>........................] - ETA: 1s - loss: 2.0010 - accuracy: 0.2726 381/1563 [======>.......................] - ETA: 1s - loss: 1.9873 - accuracy: 0.2771 419/1563 [=======>......................] - ETA: 1s - loss: 1.9789 - accuracy: 0.2800 457/1563 [=======>......................] - ETA: 1s - loss: 1.9687 - accuracy: 0.2855 495/1563 [========>.....................] - ETA: 1s - loss: 1.9555 - accuracy: 0.2902 533/1563 [=========>....................] - ETA: 1s - loss: 1.9458 - accuracy: 0.2943 571/1563 [=========>....................] - ETA: 1s - loss: 1.9356 - accuracy: 0.2987 610/1563 [==========>...................] - ETA: 1s - loss: 1.9289 - accuracy: 0.3022 650/1563 [===========>..................] - ETA: 1s - loss: 1.9223 - accuracy: 0.3051 689/1563 [============>.................] - ETA: 1s - loss: 1.9167 - accuracy: 0.3085 728/1563 [============>.................] - ETA: 1s - loss: 1.9106 - accuracy: 0.3107 766/1563 [=============>................] - ETA: 1s - loss: 1.9045 - accuracy: 0.3136 804/1563 [==============>...............] - ETA: 1s - loss: 1.8977 - accuracy: 0.3170 842/1563 [===============>..............] - ETA: 0s - loss: 1.8926 - accuracy: 0.3198 880/1563 [===============>..............] - ETA: 0s - loss: 1.8866 - accuracy: 0.3219 918/1563 [================>.............] - ETA: 0s - loss: 1.8812 - accuracy: 0.3242 955/1563 [=================>............] - ETA: 0s - loss: 1.8760 - accuracy: 0.3262 993/1563 [==================>...........] - ETA: 0s - loss: 1.8717 - accuracy: 0.3283 1031/1563 [==================>...........] - ETA: 0s - loss: 1.8665 - accuracy: 0.3300 1069/1563 [===================>..........] - ETA: 0s - loss: 1.8606 - accuracy: 0.3317 1107/1563 [====================>.........] - ETA: 0s - loss: 1.8566 - accuracy: 0.3331 1145/1563 [====================>.........] - ETA: 0s - loss: 1.8519 - accuracy: 0.3351 1183/1563 [=====================>........] - ETA: 0s - loss: 1.8470 - accuracy: 0.3372 1221/1563 [======================>.......] - ETA: 0s - loss: 1.8437 - accuracy: 0.3383 1259/1563 [=======================>......] - ETA: 0s - loss: 1.8397 - accuracy: 0.3403 1297/1563 [=======================>......] - ETA: 0s - loss: 1.8350 - accuracy: 0.3420 1335/1563 [========================>.....] - ETA: 0s - loss: 1.8313 - accuracy: 0.3434 1373/1563 [=========================>....] - ETA: 0s - loss: 1.8276 - accuracy: 0.3453 1411/1563 [==========================>...] - ETA: 0s - loss: 1.8244 - accuracy: 0.3467 1449/1563 [==========================>...] - ETA: 0s - loss: 1.8209 - accuracy: 0.3483 1487/1563 [===========================>..] - ETA: 0s - loss: 1.8173 - accuracy: 0.3496 1525/1563 [============================>.] - ETA: 0s - loss: 1.8140 - accuracy: 0.3508 1562/1563 [============================>.] - ETA: 0s - loss: 1.8118 - accuracy: 0.3522 1563/1563 [==============================] - 8s 1ms/step - loss: 1.8118 - accuracy: 0.3522 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.4996 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.6728 - accuracy: 0.4087 77/1563 [>.............................] - ETA: 1s - loss: 1.6634 - accuracy: 0.4058 115/1563 [=>............................] - ETA: 1s - loss: 1.6515 - accuracy: 0.4079 153/1563 [=>............................] - ETA: 1s - loss: 1.6498 - accuracy: 0.4118 191/1563 [==>...........................] - ETA: 1s - loss: 1.6479 - accuracy: 0.4115 229/1563 [===>..........................] - ETA: 1s - loss: 1.6494 - accuracy: 0.4117 267/1563 [====>.........................] - ETA: 1s - loss: 1.6494 - accuracy: 0.4140 305/1563 [====>.........................] - ETA: 1s - loss: 1.6494 - accuracy: 0.4155 343/1563 [=====>........................] - ETA: 1s - loss: 1.6471 - accuracy: 0.4155 381/1563 [======>.......................] - ETA: 1s - loss: 1.6486 - accuracy: 0.4145 419/1563 [=======>......................] - ETA: 1s - loss: 1.6524 - accuracy: 0.4110 457/1563 [=======>......................] - ETA: 1s - loss: 1.6523 - accuracy: 0.4121 495/1563 [========>.....................] - ETA: 1s - loss: 1.6516 - accuracy: 0.4129 533/1563 [=========>....................] - ETA: 1s - loss: 1.6485 - accuracy: 0.4154 571/1563 [=========>....................] - ETA: 1s - loss: 1.6497 - accuracy: 0.4158 609/1563 [==========>...................] - ETA: 1s - loss: 1.6487 - accuracy: 0.4170 647/1563 [===========>..................] - ETA: 1s - loss: 1.6475 - accuracy: 0.4174 685/1563 [============>.................] - ETA: 1s - loss: 1.6482 - accuracy: 0.4172 723/1563 [============>.................] - ETA: 1s - loss: 1.6472 - accuracy: 0.4181 761/1563 [=============>................] - ETA: 1s - loss: 1.6457 - accuracy: 0.4189 799/1563 [==============>...............] - ETA: 1s - loss: 1.6461 - accuracy: 0.4184 837/1563 [===============>..............] - ETA: 0s - loss: 1.6458 - accuracy: 0.4185 875/1563 [===============>..............] - ETA: 0s - loss: 1.6452 - accuracy: 0.4181 912/1563 [================>.............] - ETA: 0s - loss: 1.6431 - accuracy: 0.4188 950/1563 [=================>............] - ETA: 0s - loss: 1.6438 - accuracy: 0.4187 988/1563 [=================>............] - ETA: 0s - loss: 1.6431 - accuracy: 0.4188 1026/1563 [==================>...........] - ETA: 0s - loss: 1.6427 - accuracy: 0.4189 1064/1563 [===================>..........] - ETA: 0s - loss: 1.6431 - accuracy: 0.4191 1102/1563 [====================>.........] - ETA: 0s - loss: 1.6420 - accuracy: 0.4195 1140/1563 [====================>.........] - ETA: 0s - loss: 1.6395 - accuracy: 0.4198 1178/1563 [=====================>........] - ETA: 0s - loss: 1.6369 - accuracy: 0.4213 1216/1563 [======================>.......] - ETA: 0s - loss: 1.6345 - accuracy: 0.4224 1254/1563 [=======================>......] - ETA: 0s - loss: 1.6347 - accuracy: 0.4224 1292/1563 [=======================>......] - ETA: 0s - loss: 1.6324 - accuracy: 0.4234 1330/1563 [========================>.....] - ETA: 0s - loss: 1.6313 - accuracy: 0.4235 1368/1563 [=========================>....] - ETA: 0s - loss: 1.6289 - accuracy: 0.4247 1406/1563 [=========================>....] - ETA: 0s - loss: 1.6265 - accuracy: 0.4258 1444/1563 [==========================>...] - ETA: 0s - loss: 1.6250 - accuracy: 0.4265 1482/1563 [===========================>..] - ETA: 0s - loss: 1.6246 - accuracy: 0.4262 1520/1563 [============================>.] - ETA: 0s - loss: 1.6241 - accuracy: 0.4259 1558/1563 [============================>.] - ETA: 0s - loss: 1.6240 - accuracy: 0.4263 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6240 - accuracy: 0.4262 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.7228 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.5727 - accuracy: 0.4327 77/1563 [>.............................] - ETA: 1s - loss: 1.5642 - accuracy: 0.4375 115/1563 [=>............................] - ETA: 1s - loss: 1.5549 - accuracy: 0.4443 153/1563 [=>............................] - ETA: 1s - loss: 1.5556 - accuracy: 0.4465 191/1563 [==>...........................] - ETA: 1s - loss: 1.5473 - accuracy: 0.4496 229/1563 [===>..........................] - ETA: 1s - loss: 1.5511 - accuracy: 0.4498 267/1563 [====>.........................] - ETA: 1s - loss: 1.5466 - accuracy: 0.4524 305/1563 [====>.........................] - ETA: 1s - loss: 1.5557 - accuracy: 0.4499 343/1563 [=====>........................] - ETA: 1s - loss: 1.5537 - accuracy: 0.4507 381/1563 [======>.......................] - ETA: 1s - loss: 1.5514 - accuracy: 0.4516 419/1563 [=======>......................] - ETA: 1s - loss: 1.5496 - accuracy: 0.4523 457/1563 [=======>......................] - ETA: 1s - loss: 1.5504 - accuracy: 0.4519 495/1563 [========>.....................] - ETA: 1s - loss: 1.5538 - accuracy: 0.4509 533/1563 [=========>....................] - ETA: 1s - loss: 1.5513 - accuracy: 0.4523 571/1563 [=========>....................] - ETA: 1s - loss: 1.5482 - accuracy: 0.4538 609/1563 [==========>...................] - ETA: 1s - loss: 1.5487 - accuracy: 0.4516 647/1563 [===========>..................] - ETA: 1s - loss: 1.5497 - accuracy: 0.4519 684/1563 [============>.................] - ETA: 1s - loss: 1.5487 - accuracy: 0.4515 721/1563 [============>.................] - ETA: 1s - loss: 1.5455 - accuracy: 0.4528 758/1563 [=============>................] - ETA: 1s - loss: 1.5441 - accuracy: 0.4527 796/1563 [==============>...............] - ETA: 1s - loss: 1.5451 - accuracy: 0.4516 834/1563 [===============>..............] - ETA: 0s - loss: 1.5450 - accuracy: 0.4525 872/1563 [===============>..............] - ETA: 0s - loss: 1.5456 - accuracy: 0.4528 910/1563 [================>.............] - ETA: 0s - loss: 1.5461 - accuracy: 0.4528 948/1563 [=================>............] - ETA: 0s - loss: 1.5461 - accuracy: 0.4538 986/1563 [=================>............] - ETA: 0s - loss: 1.5458 - accuracy: 0.4549 1024/1563 [==================>...........] - ETA: 0s - loss: 1.5444 - accuracy: 0.4546 1062/1563 [===================>..........] - ETA: 0s - loss: 1.5445 - accuracy: 0.4549 1100/1563 [====================>.........] - ETA: 0s - loss: 1.5450 - accuracy: 0.4549 1138/1563 [====================>.........] - ETA: 0s - loss: 1.5436 - accuracy: 0.4547 1176/1563 [=====================>........] - ETA: 0s - loss: 1.5437 - accuracy: 0.4549 1214/1563 [======================>.......] - ETA: 0s - loss: 1.5423 - accuracy: 0.4553 1252/1563 [=======================>......] - ETA: 0s - loss: 1.5417 - accuracy: 0.4557 1290/1563 [=======================>......] - ETA: 0s - loss: 1.5423 - accuracy: 0.4552 1328/1563 [========================>.....] - ETA: 0s - loss: 1.5411 - accuracy: 0.4556 1366/1563 [=========================>....] - ETA: 0s - loss: 1.5419 - accuracy: 0.4555 1404/1563 [=========================>....] - ETA: 0s - loss: 1.5425 - accuracy: 0.4553 1442/1563 [==========================>...] - ETA: 0s - loss: 1.5425 - accuracy: 0.4556 1480/1563 [===========================>..] - ETA: 0s - loss: 1.5430 - accuracy: 0.4550 1518/1563 [============================>.] - ETA: 0s - loss: 1.5424 - accuracy: 0.4552 1556/1563 [============================>.] - ETA: 0s - loss: 1.5428 - accuracy: 0.4553 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5425 - accuracy: 0.4554 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.2497 - accuracy: 0.5625 39/1563 [..............................] - ETA: 2s - loss: 1.5134 - accuracy: 0.4671 76/1563 [>.............................] - ETA: 2s - loss: 1.5006 - accuracy: 0.4733 113/1563 [=>............................] - ETA: 1s - loss: 1.5140 - accuracy: 0.4646 151/1563 [=>............................] - ETA: 1s - loss: 1.5121 - accuracy: 0.4690 189/1563 [==>...........................] - ETA: 1s - loss: 1.5108 - accuracy: 0.4684 227/1563 [===>..........................] - ETA: 1s - loss: 1.5023 - accuracy: 0.4672 265/1563 [====>.........................] - ETA: 1s - loss: 1.4999 - accuracy: 0.4656 303/1563 [====>.........................] - ETA: 1s - loss: 1.4948 - accuracy: 0.4684 341/1563 [=====>........................] - ETA: 1s - loss: 1.4962 - accuracy: 0.4668 379/1563 [======>.......................] - ETA: 1s - loss: 1.4989 - accuracy: 0.4672 417/1563 [=======>......................] - ETA: 1s - loss: 1.4951 - accuracy: 0.4695 455/1563 [=======>......................] - ETA: 1s - loss: 1.4924 - accuracy: 0.4701 493/1563 [========>.....................] - ETA: 1s - loss: 1.4898 - accuracy: 0.4714 531/1563 [=========>....................] - ETA: 1s - loss: 1.4922 - accuracy: 0.4715 569/1563 [=========>....................] - ETA: 1s - loss: 1.4904 - accuracy: 0.4719 606/1563 [==========>...................] - ETA: 1s - loss: 1.4911 - accuracy: 0.4716 643/1563 [===========>..................] - ETA: 1s - loss: 1.4914 - accuracy: 0.4719 680/1563 [============>.................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4730 717/1563 [============>.................] - ETA: 1s - loss: 1.4888 - accuracy: 0.4725 755/1563 [=============>................] - ETA: 1s - loss: 1.4892 - accuracy: 0.4721 793/1563 [==============>...............] - ETA: 1s - loss: 1.4898 - accuracy: 0.4720 831/1563 [==============>...............] - ETA: 0s - loss: 1.4894 - accuracy: 0.4722 869/1563 [===============>..............] - ETA: 0s - loss: 1.4876 - accuracy: 0.4724 906/1563 [================>.............] - ETA: 0s - loss: 1.4881 - accuracy: 0.4722 944/1563 [=================>............] - ETA: 0s - loss: 1.4872 - accuracy: 0.4730 982/1563 [=================>............] - ETA: 0s - loss: 1.4882 - accuracy: 0.4733 1020/1563 [==================>...........] - ETA: 0s - loss: 1.4872 - accuracy: 0.4737 1058/1563 [===================>..........] - ETA: 0s - loss: 1.4870 - accuracy: 0.4738 1095/1563 [====================>.........] - ETA: 0s - loss: 1.4874 - accuracy: 0.4742 1133/1563 [====================>.........] - ETA: 0s - loss: 1.4875 - accuracy: 0.4745 1171/1563 [=====================>........] - ETA: 0s - loss: 1.4872 - accuracy: 0.4747 1208/1563 [======================>.......] - ETA: 0s - loss: 1.4867 - accuracy: 0.4744 1245/1563 [======================>.......] - ETA: 0s - loss: 1.4865 - accuracy: 0.4744 1283/1563 [=======================>......] - ETA: 0s - loss: 1.4867 - accuracy: 0.4745 1321/1563 [========================>.....] - ETA: 0s - loss: 1.4877 - accuracy: 0.4747 1359/1563 [=========================>....] - ETA: 0s - loss: 1.4877 - accuracy: 0.4751 1397/1563 [=========================>....] - ETA: 0s - loss: 1.4867 - accuracy: 0.4756 1435/1563 [==========================>...] - ETA: 0s - loss: 1.4862 - accuracy: 0.4752 1473/1563 [===========================>..] - ETA: 0s - loss: 1.4860 - accuracy: 0.4749 1511/1563 [============================>.] - ETA: 0s - loss: 1.4836 - accuracy: 0.4762 1549/1563 [============================>.] - ETA: 0s - loss: 1.4821 - accuracy: 0.4766 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4819 - accuracy: 0.4767 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.4868 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.4609 - accuracy: 0.4888 77/1563 [>.............................] - ETA: 1s - loss: 1.4837 - accuracy: 0.4842 115/1563 [=>............................] - ETA: 1s - loss: 1.4713 - accuracy: 0.4864 153/1563 [=>............................] - ETA: 1s - loss: 1.4679 - accuracy: 0.4859 191/1563 [==>...........................] - ETA: 1s - loss: 1.4619 - accuracy: 0.4864 228/1563 [===>..........................] - ETA: 1s - loss: 1.4610 - accuracy: 0.4853 265/1563 [====>.........................] - ETA: 1s - loss: 1.4566 - accuracy: 0.4868 303/1563 [====>.........................] - ETA: 1s - loss: 1.4546 - accuracy: 0.4874 340/1563 [=====>........................] - ETA: 1s - loss: 1.4546 - accuracy: 0.4875 378/1563 [======>.......................] - ETA: 1s - loss: 1.4514 - accuracy: 0.4892 416/1563 [======>.......................] - ETA: 1s - loss: 1.4458 - accuracy: 0.4905 453/1563 [=======>......................] - ETA: 1s - loss: 1.4422 - accuracy: 0.4913 491/1563 [========>.....................] - ETA: 1s - loss: 1.4417 - accuracy: 0.4914 529/1563 [=========>....................] - ETA: 1s - loss: 1.4392 - accuracy: 0.4927 564/1563 [=========>....................] - ETA: 1s - loss: 1.4397 - accuracy: 0.4930 602/1563 [==========>...................] - ETA: 1s - loss: 1.4422 - accuracy: 0.4929 639/1563 [===========>..................] - ETA: 1s - loss: 1.4424 - accuracy: 0.4926 677/1563 [===========>..................] - ETA: 1s - loss: 1.4421 - accuracy: 0.4933 715/1563 [============>.................] - ETA: 1s - loss: 1.4424 - accuracy: 0.4930 753/1563 [=============>................] - ETA: 1s - loss: 1.4418 - accuracy: 0.4929 791/1563 [==============>...............] - ETA: 1s - loss: 1.4374 - accuracy: 0.4950 829/1563 [==============>...............] - ETA: 0s - loss: 1.4372 - accuracy: 0.4940 867/1563 [===============>..............] - ETA: 0s - loss: 1.4360 - accuracy: 0.4944 905/1563 [================>.............] - ETA: 0s - loss: 1.4366 - accuracy: 0.4930 942/1563 [=================>............] - ETA: 0s - loss: 1.4370 - accuracy: 0.4927 980/1563 [=================>............] - ETA: 0s - loss: 1.4354 - accuracy: 0.4933 1018/1563 [==================>...........] - ETA: 0s - loss: 1.4347 - accuracy: 0.4934 1055/1563 [===================>..........] - ETA: 0s - loss: 1.4348 - accuracy: 0.4936 1093/1563 [===================>..........] - ETA: 0s - loss: 1.4337 - accuracy: 0.4941 1131/1563 [====================>.........] - ETA: 0s - loss: 1.4333 - accuracy: 0.4943 1169/1563 [=====================>........] - ETA: 0s - loss: 1.4332 - accuracy: 0.4945 1207/1563 [======================>.......] - ETA: 0s - loss: 1.4341 - accuracy: 0.4940 1245/1563 [======================>.......] - ETA: 0s - loss: 1.4332 - accuracy: 0.4945 1283/1563 [=======================>......] - ETA: 0s - loss: 1.4339 - accuracy: 0.4944 1321/1563 [========================>.....] - ETA: 0s - loss: 1.4330 - accuracy: 0.4946 1359/1563 [=========================>....] - ETA: 0s - loss: 1.4334 - accuracy: 0.4948 1397/1563 [=========================>....] - ETA: 0s - loss: 1.4339 - accuracy: 0.4943 1435/1563 [==========================>...] - ETA: 0s - loss: 1.4326 - accuracy: 0.4949 1473/1563 [===========================>..] - ETA: 0s - loss: 1.4335 - accuracy: 0.4945 1511/1563 [============================>.] - ETA: 0s - loss: 1.4333 - accuracy: 0.4948 1549/1563 [============================>.] - ETA: 0s - loss: 1.4336 - accuracy: 0.4944 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4336 - accuracy: 0.4944 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.4322 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.4632 - accuracy: 0.4824 76/1563 [>.............................] - ETA: 2s - loss: 1.4226 - accuracy: 0.4975 114/1563 [=>............................] - ETA: 1s - loss: 1.4029 - accuracy: 0.5055 152/1563 [=>............................] - ETA: 1s - loss: 1.3849 - accuracy: 0.5082 190/1563 [==>...........................] - ETA: 1s - loss: 1.3966 - accuracy: 0.5043 228/1563 [===>..........................] - ETA: 1s - loss: 1.3954 - accuracy: 0.5096 266/1563 [====>.........................] - ETA: 1s - loss: 1.3959 - accuracy: 0.5083 303/1563 [====>.........................] - ETA: 1s - loss: 1.3932 - accuracy: 0.5100 341/1563 [=====>........................] - ETA: 1s - loss: 1.4017 - accuracy: 0.5059 379/1563 [======>.......................] - ETA: 1s - loss: 1.3969 - accuracy: 0.5091 417/1563 [=======>......................] - ETA: 1s - loss: 1.3968 - accuracy: 0.5107 455/1563 [=======>......................] - ETA: 1s - loss: 1.3994 - accuracy: 0.5106 493/1563 [========>.....................] - ETA: 1s - loss: 1.3955 - accuracy: 0.5108 531/1563 [=========>....................] - ETA: 1s - loss: 1.3966 - accuracy: 0.5102 569/1563 [=========>....................] - ETA: 1s - loss: 1.3987 - accuracy: 0.5103 607/1563 [==========>...................] - ETA: 1s - loss: 1.3999 - accuracy: 0.5096 645/1563 [===========>..................] - ETA: 1s - loss: 1.3971 - accuracy: 0.5101 683/1563 [============>.................] - ETA: 1s - loss: 1.3969 - accuracy: 0.5099 720/1563 [============>.................] - ETA: 1s - loss: 1.3955 - accuracy: 0.5101 758/1563 [=============>................] - ETA: 1s - loss: 1.3948 - accuracy: 0.5099 796/1563 [==============>...............] - ETA: 1s - loss: 1.3950 - accuracy: 0.5099 834/1563 [===============>..............] - ETA: 0s - loss: 1.3942 - accuracy: 0.5102 872/1563 [===============>..............] - ETA: 0s - loss: 1.3932 - accuracy: 0.5113 909/1563 [================>.............] - ETA: 0s - loss: 1.3937 - accuracy: 0.5113 947/1563 [=================>............] - ETA: 0s - loss: 1.3932 - accuracy: 0.5111 985/1563 [=================>............] - ETA: 0s - loss: 1.3936 - accuracy: 0.5111 1023/1563 [==================>...........] - ETA: 0s - loss: 1.3914 - accuracy: 0.5117 1061/1563 [===================>..........] - ETA: 0s - loss: 1.3909 - accuracy: 0.5118 1099/1563 [====================>.........] - ETA: 0s - loss: 1.3913 - accuracy: 0.5118 1137/1563 [====================>.........] - ETA: 0s - loss: 1.3903 - accuracy: 0.5122 1174/1563 [=====================>........] - ETA: 0s - loss: 1.3886 - accuracy: 0.5126 1211/1563 [======================>.......] - ETA: 0s - loss: 1.3893 - accuracy: 0.5125 1248/1563 [======================>.......] - ETA: 0s - loss: 1.3885 - accuracy: 0.5128 1286/1563 [=======================>......] - ETA: 0s - loss: 1.3886 - accuracy: 0.5128 1324/1563 [========================>.....] - ETA: 0s - loss: 1.3890 - accuracy: 0.5125 1362/1563 [=========================>....] - ETA: 0s - loss: 1.3895 - accuracy: 0.5126 1400/1563 [=========================>....] - ETA: 0s - loss: 1.3891 - accuracy: 0.5127 1438/1563 [==========================>...] - ETA: 0s - loss: 1.3881 - accuracy: 0.5132 1473/1563 [===========================>..] - ETA: 0s - loss: 1.3888 - accuracy: 0.5128 1510/1563 [===========================>..] - ETA: 0s - loss: 1.3881 - accuracy: 0.5128 1548/1563 [============================>.] - ETA: 0s - loss: 1.3888 - accuracy: 0.5127 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3890 - accuracy: 0.5126 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.3252 - accuracy: 0.5625 38/1563 [..............................] - ETA: 2s - loss: 1.3660 - accuracy: 0.5058 75/1563 [>.............................] - ETA: 2s - loss: 1.3466 - accuracy: 0.5258 113/1563 [=>............................] - ETA: 1s - loss: 1.3714 - accuracy: 0.5196 151/1563 [=>............................] - ETA: 1s - loss: 1.3673 - accuracy: 0.5186 189/1563 [==>...........................] - ETA: 1s - loss: 1.3589 - accuracy: 0.5203 227/1563 [===>..........................] - ETA: 1s - loss: 1.3605 - accuracy: 0.5209 265/1563 [====>.........................] - ETA: 1s - loss: 1.3528 - accuracy: 0.5230 303/1563 [====>.........................] - ETA: 1s - loss: 1.3468 - accuracy: 0.5239 341/1563 [=====>........................] - ETA: 1s - loss: 1.3479 - accuracy: 0.5247 379/1563 [======>.......................] - ETA: 1s - loss: 1.3483 - accuracy: 0.5242 417/1563 [=======>......................] - ETA: 1s - loss: 1.3447 - accuracy: 0.5241 455/1563 [=======>......................] - ETA: 1s - loss: 1.3430 - accuracy: 0.5240 493/1563 [========>.....................] - ETA: 1s - loss: 1.3441 - accuracy: 0.5245 531/1563 [=========>....................] - ETA: 1s - loss: 1.3454 - accuracy: 0.5232 569/1563 [=========>....................] - ETA: 1s - loss: 1.3456 - accuracy: 0.5239 607/1563 [==========>...................] - ETA: 1s - loss: 1.3438 - accuracy: 0.5253 645/1563 [===========>..................] - ETA: 1s - loss: 1.3434 - accuracy: 0.5262 683/1563 [============>.................] - ETA: 1s - loss: 1.3463 - accuracy: 0.5256 721/1563 [============>.................] - ETA: 1s - loss: 1.3472 - accuracy: 0.5251 759/1563 [=============>................] - ETA: 1s - loss: 1.3451 - accuracy: 0.5263 797/1563 [==============>...............] - ETA: 1s - loss: 1.3451 - accuracy: 0.5264 835/1563 [===============>..............] - ETA: 0s - loss: 1.3438 - accuracy: 0.5268 872/1563 [===============>..............] - ETA: 0s - loss: 1.3445 - accuracy: 0.5262 910/1563 [================>.............] - ETA: 0s - loss: 1.3445 - accuracy: 0.5260 947/1563 [=================>............] - ETA: 0s - loss: 1.3445 - accuracy: 0.5267 985/1563 [=================>............] - ETA: 0s - loss: 1.3441 - accuracy: 0.5267 1023/1563 [==================>...........] - ETA: 0s - loss: 1.3459 - accuracy: 0.5258 1061/1563 [===================>..........] - ETA: 0s - loss: 1.3454 - accuracy: 0.5262 1099/1563 [====================>.........] - ETA: 0s - loss: 1.3467 - accuracy: 0.5265 1137/1563 [====================>.........] - ETA: 0s - loss: 1.3472 - accuracy: 0.5270 1175/1563 [=====================>........] - ETA: 0s - loss: 1.3475 - accuracy: 0.5270 1213/1563 [======================>.......] - ETA: 0s - loss: 1.3479 - accuracy: 0.5269 1251/1563 [=======================>......] - ETA: 0s - loss: 1.3485 - accuracy: 0.5263 1288/1563 [=======================>......] - ETA: 0s - loss: 1.3481 - accuracy: 0.5268 1326/1563 [========================>.....] - ETA: 0s - loss: 1.3481 - accuracy: 0.5262 1364/1563 [=========================>....] - ETA: 0s - loss: 1.3492 - accuracy: 0.5258 1402/1563 [=========================>....] - ETA: 0s - loss: 1.3498 - accuracy: 0.5253 1440/1563 [==========================>...] - ETA: 0s - loss: 1.3513 - accuracy: 0.5249 1476/1563 [===========================>..] - ETA: 0s - loss: 1.3507 - accuracy: 0.5251 1514/1563 [============================>.] - ETA: 0s - loss: 1.3493 - accuracy: 0.5256 1552/1563 [============================>.] - ETA: 0s - loss: 1.3486 - accuracy: 0.5260 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3485 - accuracy: 0.5262 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.2186 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.3463 - accuracy: 0.5321 77/1563 [>.............................] - ETA: 1s - loss: 1.3316 - accuracy: 0.5386 115/1563 [=>............................] - ETA: 1s - loss: 1.3119 - accuracy: 0.5459 153/1563 [=>............................] - ETA: 1s - loss: 1.3146 - accuracy: 0.5419 191/1563 [==>...........................] - ETA: 1s - loss: 1.3181 - accuracy: 0.5416 229/1563 [===>..........................] - ETA: 1s - loss: 1.3137 - accuracy: 0.5403 267/1563 [====>.........................] - ETA: 1s - loss: 1.3143 - accuracy: 0.5383 305/1563 [====>.........................] - ETA: 1s - loss: 1.3189 - accuracy: 0.5376 343/1563 [=====>........................] - ETA: 1s - loss: 1.3186 - accuracy: 0.5385 381/1563 [======>.......................] - ETA: 1s - loss: 1.3162 - accuracy: 0.5372 419/1563 [=======>......................] - ETA: 1s - loss: 1.3144 - accuracy: 0.5374 457/1563 [=======>......................] - ETA: 1s - loss: 1.3158 - accuracy: 0.5370 494/1563 [========>.....................] - ETA: 1s - loss: 1.3207 - accuracy: 0.5345 531/1563 [=========>....................] - ETA: 1s - loss: 1.3190 - accuracy: 0.5343 569/1563 [=========>....................] - ETA: 1s - loss: 1.3201 - accuracy: 0.5347 606/1563 [==========>...................] - ETA: 1s - loss: 1.3168 - accuracy: 0.5363 644/1563 [===========>..................] - ETA: 1s - loss: 1.3193 - accuracy: 0.5351 682/1563 [============>.................] - ETA: 1s - loss: 1.3197 - accuracy: 0.5345 720/1563 [============>.................] - ETA: 1s - loss: 1.3219 - accuracy: 0.5334 758/1563 [=============>................] - ETA: 1s - loss: 1.3203 - accuracy: 0.5338 796/1563 [==============>...............] - ETA: 1s - loss: 1.3224 - accuracy: 0.5338 834/1563 [===============>..............] - ETA: 0s - loss: 1.3223 - accuracy: 0.5341 872/1563 [===============>..............] - ETA: 0s - loss: 1.3221 - accuracy: 0.5338 910/1563 [================>.............] - ETA: 0s - loss: 1.3190 - accuracy: 0.5353 948/1563 [=================>............] - ETA: 0s - loss: 1.3170 - accuracy: 0.5354 986/1563 [=================>............] - ETA: 0s - loss: 1.3173 - accuracy: 0.5351 1024/1563 [==================>...........] - ETA: 0s - loss: 1.3166 - accuracy: 0.5356 1062/1563 [===================>..........] - ETA: 0s - loss: 1.3157 - accuracy: 0.5365 1100/1563 [====================>.........] - ETA: 0s - loss: 1.3152 - accuracy: 0.5370 1138/1563 [====================>.........] - ETA: 0s - loss: 1.3157 - accuracy: 0.5369 1176/1563 [=====================>........] - ETA: 0s - loss: 1.3151 - accuracy: 0.5377 1214/1563 [======================>.......] - ETA: 0s - loss: 1.3148 - accuracy: 0.5385 1252/1563 [=======================>......] - ETA: 0s - loss: 1.3157 - accuracy: 0.5378 1290/1563 [=======================>......] - ETA: 0s - loss: 1.3132 - accuracy: 0.5387 1328/1563 [========================>.....] - ETA: 0s - loss: 1.3132 - accuracy: 0.5384 1366/1563 [=========================>....] - ETA: 0s - loss: 1.3125 - accuracy: 0.5382 1404/1563 [=========================>....] - ETA: 0s - loss: 1.3148 - accuracy: 0.5378 1442/1563 [==========================>...] - ETA: 0s - loss: 1.3148 - accuracy: 0.5375 1480/1563 [===========================>..] - ETA: 0s - loss: 1.3140 - accuracy: 0.5378 1518/1563 [============================>.] - ETA: 0s - loss: 1.3151 - accuracy: 0.5376 1556/1563 [============================>.] - ETA: 0s - loss: 1.3164 - accuracy: 0.5373 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3165 - accuracy: 0.5373 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.5293 - accuracy: 0.4375 39/1563 [..............................] - ETA: 2s - loss: 1.2910 - accuracy: 0.5449 77/1563 [>.............................] - ETA: 1s - loss: 1.2972 - accuracy: 0.5463 115/1563 [=>............................] - ETA: 1s - loss: 1.3142 - accuracy: 0.5405 153/1563 [=>............................] - ETA: 1s - loss: 1.3160 - accuracy: 0.5404 191/1563 [==>...........................] - ETA: 1s - loss: 1.3056 - accuracy: 0.5456 228/1563 [===>..........................] - ETA: 1s - loss: 1.3002 - accuracy: 0.5461 265/1563 [====>.........................] - ETA: 1s - loss: 1.2986 - accuracy: 0.5447 302/1563 [====>.........................] - ETA: 1s - loss: 1.2985 - accuracy: 0.5446 340/1563 [=====>........................] - ETA: 1s - loss: 1.3025 - accuracy: 0.5426 378/1563 [======>.......................] - ETA: 1s - loss: 1.3002 - accuracy: 0.5449 416/1563 [======>.......................] - ETA: 1s - loss: 1.2959 - accuracy: 0.5460 453/1563 [=======>......................] - ETA: 1s - loss: 1.2923 - accuracy: 0.5462 491/1563 [========>.....................] - ETA: 1s - loss: 1.2919 - accuracy: 0.5469 529/1563 [=========>....................] - ETA: 1s - loss: 1.2869 - accuracy: 0.5486 567/1563 [=========>....................] - ETA: 1s - loss: 1.2898 - accuracy: 0.5476 605/1563 [==========>...................] - ETA: 1s - loss: 1.2888 - accuracy: 0.5478 643/1563 [===========>..................] - ETA: 1s - loss: 1.2862 - accuracy: 0.5481 681/1563 [============>.................] - ETA: 1s - loss: 1.2840 - accuracy: 0.5489 719/1563 [============>.................] - ETA: 1s - loss: 1.2846 - accuracy: 0.5489 757/1563 [=============>................] - ETA: 1s - loss: 1.2858 - accuracy: 0.5492 795/1563 [==============>...............] - ETA: 1s - loss: 1.2828 - accuracy: 0.5506 833/1563 [==============>...............] - ETA: 0s - loss: 1.2826 - accuracy: 0.5512 871/1563 [===============>..............] - ETA: 0s - loss: 1.2823 - accuracy: 0.5510 909/1563 [================>.............] - ETA: 0s - loss: 1.2802 - accuracy: 0.5514 947/1563 [=================>............] - ETA: 0s - loss: 1.2808 - accuracy: 0.5512 985/1563 [=================>............] - ETA: 0s - loss: 1.2802 - accuracy: 0.5515 1023/1563 [==================>...........] - ETA: 0s - loss: 1.2798 - accuracy: 0.5523 1061/1563 [===================>..........] - ETA: 0s - loss: 1.2822 - accuracy: 0.5510 1099/1563 [====================>.........] - ETA: 0s - loss: 1.2818 - accuracy: 0.5512 1137/1563 [====================>.........] - ETA: 0s - loss: 1.2818 - accuracy: 0.5508 1175/1563 [=====================>........] - ETA: 0s - loss: 1.2820 - accuracy: 0.5506 1213/1563 [======================>.......] - ETA: 0s - loss: 1.2847 - accuracy: 0.5499 1251/1563 [=======================>......] - ETA: 0s - loss: 1.2857 - accuracy: 0.5496 1289/1563 [=======================>......] - ETA: 0s - loss: 1.2843 - accuracy: 0.5500 1327/1563 [========================>.....] - ETA: 0s - loss: 1.2837 - accuracy: 0.5502 1365/1563 [=========================>....] - ETA: 0s - loss: 1.2839 - accuracy: 0.5495 1403/1563 [=========================>....] - ETA: 0s - loss: 1.2832 - accuracy: 0.5496 1441/1563 [==========================>...] - ETA: 0s - loss: 1.2832 - accuracy: 0.5498 1478/1563 [===========================>..] - ETA: 0s - loss: 1.2835 - accuracy: 0.5496 1516/1563 [============================>.] - ETA: 0s - loss: 1.2848 - accuracy: 0.5493 1554/1563 [============================>.] - ETA: 0s - loss: 1.2852 - accuracy: 0.5492 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2853 - accuracy: 0.5492 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.1180 - accuracy: 0.5625 39/1563 [..............................] - ETA: 2s - loss: 1.2569 - accuracy: 0.5385 77/1563 [>.............................] - ETA: 1s - loss: 1.2752 - accuracy: 0.5394 114/1563 [=>............................] - ETA: 1s - loss: 1.2778 - accuracy: 0.5430 152/1563 [=>............................] - ETA: 1s - loss: 1.2720 - accuracy: 0.5448 190/1563 [==>...........................] - ETA: 1s - loss: 1.2722 - accuracy: 0.5457 228/1563 [===>..........................] - ETA: 1s - loss: 1.2685 - accuracy: 0.5503 266/1563 [====>.........................] - ETA: 1s - loss: 1.2599 - accuracy: 0.5556 303/1563 [====>.........................] - ETA: 1s - loss: 1.2588 - accuracy: 0.5571 340/1563 [=====>........................] - ETA: 1s - loss: 1.2572 - accuracy: 0.5582 378/1563 [======>.......................] - ETA: 1s - loss: 1.2540 - accuracy: 0.5583 416/1563 [======>.......................] - ETA: 1s - loss: 1.2532 - accuracy: 0.5591 453/1563 [=======>......................] - ETA: 1s - loss: 1.2496 - accuracy: 0.5607 491/1563 [========>.....................] - ETA: 1s - loss: 1.2496 - accuracy: 0.5617 528/1563 [=========>....................] - ETA: 1s - loss: 1.2461 - accuracy: 0.5620 565/1563 [=========>....................] - ETA: 1s - loss: 1.2473 - accuracy: 0.5613 603/1563 [==========>...................] - ETA: 1s - loss: 1.2506 - accuracy: 0.5605 641/1563 [===========>..................] - ETA: 1s - loss: 1.2485 - accuracy: 0.5616 679/1563 [============>.................] - ETA: 1s - loss: 1.2495 - accuracy: 0.5611 717/1563 [============>.................] - ETA: 1s - loss: 1.2507 - accuracy: 0.5608 755/1563 [=============>................] - ETA: 1s - loss: 1.2484 - accuracy: 0.5622 793/1563 [==============>...............] - ETA: 1s - loss: 1.2490 - accuracy: 0.5622 831/1563 [==============>...............] - ETA: 0s - loss: 1.2505 - accuracy: 0.5622 869/1563 [===============>..............] - ETA: 0s - loss: 1.2498 - accuracy: 0.5625 907/1563 [================>.............] - ETA: 0s - loss: 1.2499 - accuracy: 0.5625 945/1563 [=================>............] - ETA: 0s - loss: 1.2514 - accuracy: 0.5618 983/1563 [=================>............] - ETA: 0s - loss: 1.2505 - accuracy: 0.5620 1020/1563 [==================>...........] - ETA: 0s - loss: 1.2492 - accuracy: 0.5625 1057/1563 [===================>..........] - ETA: 0s - loss: 1.2479 - accuracy: 0.5632 1095/1563 [====================>.........] - ETA: 0s - loss: 1.2479 - accuracy: 0.5634 1133/1563 [====================>.........] - ETA: 0s - loss: 1.2488 - accuracy: 0.5635 1171/1563 [=====================>........] - ETA: 0s - loss: 1.2470 - accuracy: 0.5640 1208/1563 [======================>.......] - ETA: 0s - loss: 1.2478 - accuracy: 0.5640 1245/1563 [======================>.......] - ETA: 0s - loss: 1.2489 - accuracy: 0.5635 1283/1563 [=======================>......] - ETA: 0s - loss: 1.2500 - accuracy: 0.5624 1321/1563 [========================>.....] - ETA: 0s - loss: 1.2502 - accuracy: 0.5624 1359/1563 [=========================>....] - ETA: 0s - loss: 1.2513 - accuracy: 0.5623 1397/1563 [=========================>....] - ETA: 0s - loss: 1.2538 - accuracy: 0.5611 1435/1563 [==========================>...] - ETA: 0s - loss: 1.2549 - accuracy: 0.5610 1473/1563 [===========================>..] - ETA: 0s - loss: 1.2545 - accuracy: 0.5612 1511/1563 [============================>.] - ETA: 0s - loss: 1.2551 - accuracy: 0.5611 1549/1563 [============================>.] - ETA: 0s - loss: 1.2546 - accuracy: 0.5614 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2546 - accuracy: 0.5617 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 45.8397376537323 hydro03 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins15382284926932086226.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS