Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 98520 queued and waiting for resources srun: job 98520 has been allocated resources 2024-11-24 07:43:02.918527: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-24 07:43:03.058755: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-24 07:43:04.034566: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-24 07:43:04.034642: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-24 07:43:04.034649: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-24 07:43:07.104083: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-24 07:43:07.478109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-24 07:43:09.548611: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 27:59 - loss: 2.3143 - accuracy: 0.0938 37/1563 [..............................] - ETA: 2s - loss: 2.2484 - accuracy: 0.1495  75/1563 [>.............................] - ETA: 2s - loss: 2.1651 - accuracy: 0.1879 113/1563 [=>............................] - ETA: 1s - loss: 2.1094 - accuracy: 0.2185 151/1563 [=>............................] - ETA: 1s - loss: 2.0850 - accuracy: 0.2347 189/1563 [==>...........................] - ETA: 1s - loss: 2.0586 - accuracy: 0.2507 227/1563 [===>..........................] - ETA: 1s - loss: 2.0400 - accuracy: 0.2596 265/1563 [====>.........................] - ETA: 1s - loss: 2.0162 - accuracy: 0.2696 303/1563 [====>.........................] - ETA: 1s - loss: 2.0011 - accuracy: 0.2751 341/1563 [=====>........................] - ETA: 1s - loss: 1.9868 - accuracy: 0.2809 379/1563 [======>.......................] - ETA: 1s - loss: 1.9770 - accuracy: 0.2844 417/1563 [=======>......................] - ETA: 1s - loss: 1.9664 - accuracy: 0.2908 455/1563 [=======>......................] - ETA: 1s - loss: 1.9599 - accuracy: 0.2946 493/1563 [========>.....................] - ETA: 1s - loss: 1.9481 - accuracy: 0.2993 531/1563 [=========>....................] - ETA: 1s - loss: 1.9411 - accuracy: 0.3018 569/1563 [=========>....................] - ETA: 1s - loss: 1.9312 - accuracy: 0.3070 607/1563 [==========>...................] - ETA: 1s - loss: 1.9234 - accuracy: 0.3107 645/1563 [===========>..................] - ETA: 1s - loss: 1.9176 - accuracy: 0.3140 683/1563 [============>.................] - ETA: 1s - loss: 1.9110 - accuracy: 0.3175 721/1563 [============>.................] - ETA: 1s - loss: 1.9064 - accuracy: 0.3192 759/1563 [=============>................] - ETA: 1s - loss: 1.8987 - accuracy: 0.3214 797/1563 [==============>...............] - ETA: 1s - loss: 1.8934 - accuracy: 0.3238 835/1563 [===============>..............] - ETA: 0s - loss: 1.8876 - accuracy: 0.3260 873/1563 [===============>..............] - ETA: 0s - loss: 1.8818 - accuracy: 0.3280 911/1563 [================>.............] - ETA: 0s - loss: 1.8748 - accuracy: 0.3304 949/1563 [=================>............] - ETA: 0s - loss: 1.8701 - accuracy: 0.3322 987/1563 [=================>............] - ETA: 0s - loss: 1.8659 - accuracy: 0.3341 1025/1563 [==================>...........] - ETA: 0s - loss: 1.8620 - accuracy: 0.3358 1063/1563 [===================>..........] - ETA: 0s - loss: 1.8586 - accuracy: 0.3372 1101/1563 [====================>.........] - ETA: 0s - loss: 1.8550 - accuracy: 0.3385 1139/1563 [====================>.........] - ETA: 0s - loss: 1.8503 - accuracy: 0.3400 1177/1563 [=====================>........] - ETA: 0s - loss: 1.8460 - accuracy: 0.3414 1215/1563 [======================>.......] - ETA: 0s - loss: 1.8437 - accuracy: 0.3421 1253/1563 [=======================>......] - ETA: 0s - loss: 1.8388 - accuracy: 0.3441 1291/1563 [=======================>......] - ETA: 0s - loss: 1.8355 - accuracy: 0.3454 1329/1563 [========================>.....] - ETA: 0s - loss: 1.8323 - accuracy: 0.3462 1367/1563 [=========================>....] - ETA: 0s - loss: 1.8277 - accuracy: 0.3478 1405/1563 [=========================>....] - ETA: 0s - loss: 1.8235 - accuracy: 0.3496 1443/1563 [==========================>...] - ETA: 0s - loss: 1.8202 - accuracy: 0.3513 1481/1563 [===========================>..] - ETA: 0s - loss: 1.8169 - accuracy: 0.3531 1519/1563 [============================>.] - ETA: 0s - loss: 1.8139 - accuracy: 0.3541 1557/1563 [============================>.] - ETA: 0s - loss: 1.8102 - accuracy: 0.3552 1563/1563 [==============================] - 3s 1ms/step - loss: 1.8097 - accuracy: 0.3554 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.9048 - accuracy: 0.2812 39/1563 [..............................] - ETA: 2s - loss: 1.6667 - accuracy: 0.4030 77/1563 [>.............................] - ETA: 1s - loss: 1.6688 - accuracy: 0.4119 115/1563 [=>............................] - ETA: 1s - loss: 1.6614 - accuracy: 0.4139 153/1563 [=>............................] - ETA: 1s - loss: 1.6516 - accuracy: 0.4158 191/1563 [==>...........................] - ETA: 1s - loss: 1.6530 - accuracy: 0.4190 229/1563 [===>..........................] - ETA: 1s - loss: 1.6469 - accuracy: 0.4225 267/1563 [====>.........................] - ETA: 1s - loss: 1.6419 - accuracy: 0.4247 305/1563 [====>.........................] - ETA: 1s - loss: 1.6464 - accuracy: 0.4247 343/1563 [=====>........................] - ETA: 1s - loss: 1.6429 - accuracy: 0.4244 380/1563 [======>.......................] - ETA: 1s - loss: 1.6409 - accuracy: 0.4252 418/1563 [=======>......................] - ETA: 1s - loss: 1.6397 - accuracy: 0.4262 456/1563 [=======>......................] - ETA: 1s - loss: 1.6378 - accuracy: 0.4252 494/1563 [========>.....................] - ETA: 1s - loss: 1.6370 - accuracy: 0.4247 532/1563 [=========>....................] - ETA: 1s - loss: 1.6348 - accuracy: 0.4253 570/1563 [=========>....................] - ETA: 1s - loss: 1.6346 - accuracy: 0.4248 608/1563 [==========>...................] - ETA: 1s - loss: 1.6332 - accuracy: 0.4251 646/1563 [===========>..................] - ETA: 1s - loss: 1.6336 - accuracy: 0.4257 684/1563 [============>.................] - ETA: 1s - loss: 1.6337 - accuracy: 0.4254 722/1563 [============>.................] - ETA: 1s - loss: 1.6334 - accuracy: 0.4257 760/1563 [=============>................] - ETA: 1s - loss: 1.6325 - accuracy: 0.4260 798/1563 [==============>...............] - ETA: 1s - loss: 1.6315 - accuracy: 0.4266 835/1563 [===============>..............] - ETA: 0s - loss: 1.6320 - accuracy: 0.4267 872/1563 [===============>..............] - ETA: 0s - loss: 1.6325 - accuracy: 0.4269 910/1563 [================>.............] - ETA: 0s - loss: 1.6302 - accuracy: 0.4273 948/1563 [=================>............] - ETA: 0s - loss: 1.6291 - accuracy: 0.4276 986/1563 [=================>............] - ETA: 0s - loss: 1.6293 - accuracy: 0.4277 1024/1563 [==================>...........] - ETA: 0s - loss: 1.6304 - accuracy: 0.4274 1062/1563 [===================>..........] - ETA: 0s - loss: 1.6296 - accuracy: 0.4278 1100/1563 [====================>.........] - ETA: 0s - loss: 1.6304 - accuracy: 0.4272 1138/1563 [====================>.........] - ETA: 0s - loss: 1.6296 - accuracy: 0.4269 1176/1563 [=====================>........] - ETA: 0s - loss: 1.6278 - accuracy: 0.4275 1214/1563 [======================>.......] - ETA: 0s - loss: 1.6265 - accuracy: 0.4281 1252/1563 [=======================>......] - ETA: 0s - loss: 1.6263 - accuracy: 0.4281 1290/1563 [=======================>......] - ETA: 0s - loss: 1.6268 - accuracy: 0.4279 1328/1563 [========================>.....] - ETA: 0s - loss: 1.6257 - accuracy: 0.4277 1366/1563 [=========================>....] - ETA: 0s - loss: 1.6239 - accuracy: 0.4284 1404/1563 [=========================>....] - ETA: 0s - loss: 1.6238 - accuracy: 0.4280 1442/1563 [==========================>...] - ETA: 0s - loss: 1.6225 - accuracy: 0.4286 1480/1563 [===========================>..] - ETA: 0s - loss: 1.6219 - accuracy: 0.4285 1518/1563 [============================>.] - ETA: 0s - loss: 1.6212 - accuracy: 0.4285 1555/1563 [============================>.] - ETA: 0s - loss: 1.6205 - accuracy: 0.4288 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6204 - accuracy: 0.4288 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.8260 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.5438 - accuracy: 0.4471 77/1563 [>.............................] - ETA: 1s - loss: 1.5558 - accuracy: 0.4476 115/1563 [=>............................] - ETA: 1s - loss: 1.5661 - accuracy: 0.4443 153/1563 [=>............................] - ETA: 1s - loss: 1.5697 - accuracy: 0.4446 191/1563 [==>...........................] - ETA: 1s - loss: 1.5679 - accuracy: 0.4437 229/1563 [===>..........................] - ETA: 1s - loss: 1.5678 - accuracy: 0.4434 267/1563 [====>.........................] - ETA: 1s - loss: 1.5594 - accuracy: 0.4462 305/1563 [====>.........................] - ETA: 1s - loss: 1.5634 - accuracy: 0.4450 343/1563 [=====>........................] - ETA: 1s - loss: 1.5658 - accuracy: 0.4442 381/1563 [======>.......................] - ETA: 1s - loss: 1.5709 - accuracy: 0.4432 419/1563 [=======>......................] - ETA: 1s - loss: 1.5677 - accuracy: 0.4433 457/1563 [=======>......................] - ETA: 1s - loss: 1.5638 - accuracy: 0.4452 495/1563 [========>.....................] - ETA: 1s - loss: 1.5626 - accuracy: 0.4465 533/1563 [=========>....................] - ETA: 1s - loss: 1.5613 - accuracy: 0.4482 571/1563 [=========>....................] - ETA: 1s - loss: 1.5583 - accuracy: 0.4493 609/1563 [==========>...................] - ETA: 1s - loss: 1.5624 - accuracy: 0.4481 647/1563 [===========>..................] - ETA: 1s - loss: 1.5634 - accuracy: 0.4480 685/1563 [============>.................] - ETA: 1s - loss: 1.5597 - accuracy: 0.4494 723/1563 [============>.................] - ETA: 1s - loss: 1.5585 - accuracy: 0.4498 761/1563 [=============>................] - ETA: 1s - loss: 1.5583 - accuracy: 0.4506 799/1563 [==============>...............] - ETA: 1s - loss: 1.5568 - accuracy: 0.4511 837/1563 [===============>..............] - ETA: 0s - loss: 1.5550 - accuracy: 0.4508 875/1563 [===============>..............] - ETA: 0s - loss: 1.5535 - accuracy: 0.4505 913/1563 [================>.............] - ETA: 0s - loss: 1.5532 - accuracy: 0.4503 951/1563 [=================>............] - ETA: 0s - loss: 1.5540 - accuracy: 0.4503 989/1563 [=================>............] - ETA: 0s - loss: 1.5526 - accuracy: 0.4505 1027/1563 [==================>...........] - ETA: 0s - loss: 1.5523 - accuracy: 0.4505 1065/1563 [===================>..........] - ETA: 0s - loss: 1.5506 - accuracy: 0.4510 1103/1563 [====================>.........] - ETA: 0s - loss: 1.5503 - accuracy: 0.4518 1141/1563 [====================>.........] - ETA: 0s - loss: 1.5493 - accuracy: 0.4521 1179/1563 [=====================>........] - ETA: 0s - loss: 1.5484 - accuracy: 0.4527 1217/1563 [======================>.......] - ETA: 0s - loss: 1.5489 - accuracy: 0.4525 1255/1563 [=======================>......] - ETA: 0s - loss: 1.5465 - accuracy: 0.4533 1292/1563 [=======================>......] - ETA: 0s - loss: 1.5463 - accuracy: 0.4536 1330/1563 [========================>.....] - ETA: 0s - loss: 1.5451 - accuracy: 0.4537 1368/1563 [=========================>....] - ETA: 0s - loss: 1.5432 - accuracy: 0.4546 1406/1563 [=========================>....] - ETA: 0s - loss: 1.5439 - accuracy: 0.4544 1444/1563 [==========================>...] - ETA: 0s - loss: 1.5423 - accuracy: 0.4554 1482/1563 [===========================>..] - ETA: 0s - loss: 1.5421 - accuracy: 0.4552 1520/1563 [============================>.] - ETA: 0s - loss: 1.5413 - accuracy: 0.4557 1558/1563 [============================>.] - ETA: 0s - loss: 1.5417 - accuracy: 0.4555 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5414 - accuracy: 0.4556 Epoch 4/10 1/1563 [..............................] - ETA: 2s - loss: 1.6403 - accuracy: 0.4688 39/1563 [..............................] - ETA: 2s - loss: 1.4923 - accuracy: 0.4968 77/1563 [>.............................] - ETA: 1s - loss: 1.4667 - accuracy: 0.5000 115/1563 [=>............................] - ETA: 1s - loss: 1.4899 - accuracy: 0.4894 153/1563 [=>............................] - ETA: 1s - loss: 1.4856 - accuracy: 0.4882 191/1563 [==>...........................] - ETA: 1s - loss: 1.4897 - accuracy: 0.4843 229/1563 [===>..........................] - ETA: 1s - loss: 1.4887 - accuracy: 0.4853 266/1563 [====>.........................] - ETA: 1s - loss: 1.4905 - accuracy: 0.4836 303/1563 [====>.........................] - ETA: 1s - loss: 1.4854 - accuracy: 0.4856 341/1563 [=====>........................] - ETA: 1s - loss: 1.4838 - accuracy: 0.4834 379/1563 [======>.......................] - ETA: 1s - loss: 1.4840 - accuracy: 0.4824 417/1563 [=======>......................] - ETA: 1s - loss: 1.4865 - accuracy: 0.4809 455/1563 [=======>......................] - ETA: 1s - loss: 1.4880 - accuracy: 0.4799 493/1563 [========>.....................] - ETA: 1s - loss: 1.4895 - accuracy: 0.4787 531/1563 [=========>....................] - ETA: 1s - loss: 1.4847 - accuracy: 0.4797 569/1563 [=========>....................] - ETA: 1s - loss: 1.4823 - accuracy: 0.4796 607/1563 [==========>...................] - ETA: 1s - loss: 1.4832 - accuracy: 0.4790 645/1563 [===========>..................] - ETA: 1s - loss: 1.4830 - accuracy: 0.4795 683/1563 [============>.................] - ETA: 1s - loss: 1.4816 - accuracy: 0.4798 721/1563 [============>.................] - ETA: 1s - loss: 1.4824 - accuracy: 0.4792 759/1563 [=============>................] - ETA: 1s - loss: 1.4826 - accuracy: 0.4783 797/1563 [==============>...............] - ETA: 1s - loss: 1.4845 - accuracy: 0.4775 835/1563 [===============>..............] - ETA: 0s - loss: 1.4845 - accuracy: 0.4773 873/1563 [===============>..............] - ETA: 0s - loss: 1.4846 - accuracy: 0.4777 911/1563 [================>.............] - ETA: 0s - loss: 1.4835 - accuracy: 0.4778 949/1563 [=================>............] - ETA: 0s - loss: 1.4829 - accuracy: 0.4781 987/1563 [=================>............] - ETA: 0s - loss: 1.4805 - accuracy: 0.4790 1025/1563 [==================>...........] - ETA: 0s - loss: 1.4803 - accuracy: 0.4789 1063/1563 [===================>..........] - ETA: 0s - loss: 1.4822 - accuracy: 0.4792 1101/1563 [====================>.........] - ETA: 0s - loss: 1.4810 - accuracy: 0.4793 1139/1563 [====================>.........] - ETA: 0s - loss: 1.4796 - accuracy: 0.4793 1177/1563 [=====================>........] - ETA: 0s - loss: 1.4810 - accuracy: 0.4784 1215/1563 [======================>.......] - ETA: 0s - loss: 1.4810 - accuracy: 0.4778 1253/1563 [=======================>......] - ETA: 0s - loss: 1.4824 - accuracy: 0.4773 1291/1563 [=======================>......] - ETA: 0s - loss: 1.4816 - accuracy: 0.4774 1329/1563 [========================>.....] - ETA: 0s - loss: 1.4811 - accuracy: 0.4777 1367/1563 [=========================>....] - ETA: 0s - loss: 1.4816 - accuracy: 0.4773 1405/1563 [=========================>....] - ETA: 0s - loss: 1.4816 - accuracy: 0.4775 1443/1563 [==========================>...] - ETA: 0s - loss: 1.4801 - accuracy: 0.4785 1481/1563 [===========================>..] - ETA: 0s - loss: 1.4796 - accuracy: 0.4786 1519/1563 [============================>.] - ETA: 0s - loss: 1.4802 - accuracy: 0.4789 1557/1563 [============================>.] - ETA: 0s - loss: 1.4794 - accuracy: 0.4790 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4793 - accuracy: 0.4790 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.9068 - accuracy: 0.2812 39/1563 [..............................] - ETA: 2s - loss: 1.4710 - accuracy: 0.4720 77/1563 [>.............................] - ETA: 1s - loss: 1.4552 - accuracy: 0.4773 115/1563 [=>............................] - ETA: 1s - loss: 1.4328 - accuracy: 0.4965 153/1563 [=>............................] - ETA: 1s - loss: 1.4305 - accuracy: 0.4945 191/1563 [==>...........................] - ETA: 1s - loss: 1.4317 - accuracy: 0.4964 229/1563 [===>..........................] - ETA: 1s - loss: 1.4330 - accuracy: 0.4939 267/1563 [====>.........................] - ETA: 1s - loss: 1.4330 - accuracy: 0.4926 304/1563 [====>.........................] - ETA: 1s - loss: 1.4357 - accuracy: 0.4914 341/1563 [=====>........................] - ETA: 1s - loss: 1.4331 - accuracy: 0.4940 379/1563 [======>.......................] - ETA: 1s - loss: 1.4322 - accuracy: 0.4958 417/1563 [=======>......................] - ETA: 1s - loss: 1.4316 - accuracy: 0.4951 455/1563 [=======>......................] - ETA: 1s - loss: 1.4286 - accuracy: 0.4977 493/1563 [========>.....................] - ETA: 1s - loss: 1.4283 - accuracy: 0.4974 531/1563 [=========>....................] - ETA: 1s - loss: 1.4300 - accuracy: 0.4955 569/1563 [=========>....................] - ETA: 1s - loss: 1.4282 - accuracy: 0.4953 607/1563 [==========>...................] - ETA: 1s - loss: 1.4326 - accuracy: 0.4943 645/1563 [===========>..................] - ETA: 1s - loss: 1.4318 - accuracy: 0.4945 682/1563 [============>.................] - ETA: 1s - loss: 1.4315 - accuracy: 0.4941 720/1563 [============>.................] - ETA: 1s - loss: 1.4322 - accuracy: 0.4947 758/1563 [=============>................] - ETA: 1s - loss: 1.4337 - accuracy: 0.4937 796/1563 [==============>...............] - ETA: 1s - loss: 1.4321 - accuracy: 0.4947 834/1563 [===============>..............] - ETA: 0s - loss: 1.4324 - accuracy: 0.4946 872/1563 [===============>..............] - ETA: 0s - loss: 1.4329 - accuracy: 0.4943 909/1563 [================>.............] - ETA: 0s - loss: 1.4322 - accuracy: 0.4945 947/1563 [=================>............] - ETA: 0s - loss: 1.4326 - accuracy: 0.4946 985/1563 [=================>............] - ETA: 0s - loss: 1.4334 - accuracy: 0.4946 1023/1563 [==================>...........] - ETA: 0s - loss: 1.4338 - accuracy: 0.4946 1061/1563 [===================>..........] - ETA: 0s - loss: 1.4335 - accuracy: 0.4947 1099/1563 [====================>.........] - ETA: 0s - loss: 1.4330 - accuracy: 0.4950 1137/1563 [====================>.........] - ETA: 0s - loss: 1.4322 - accuracy: 0.4951 1175/1563 [=====================>........] - ETA: 0s - loss: 1.4322 - accuracy: 0.4948 1213/1563 [======================>.......] - ETA: 0s - loss: 1.4326 - accuracy: 0.4950 1251/1563 [=======================>......] - ETA: 0s - loss: 1.4316 - accuracy: 0.4954 1289/1563 [=======================>......] - ETA: 0s - loss: 1.4321 - accuracy: 0.4956 1327/1563 [========================>.....] - ETA: 0s - loss: 1.4316 - accuracy: 0.4955 1365/1563 [=========================>....] - ETA: 0s - loss: 1.4303 - accuracy: 0.4956 1403/1563 [=========================>....] - ETA: 0s - loss: 1.4311 - accuracy: 0.4951 1441/1563 [==========================>...] - ETA: 0s - loss: 1.4313 - accuracy: 0.4951 1479/1563 [===========================>..] - ETA: 0s - loss: 1.4316 - accuracy: 0.4956 1516/1563 [============================>.] - ETA: 0s - loss: 1.4312 - accuracy: 0.4960 1553/1563 [============================>.] - ETA: 0s - loss: 1.4304 - accuracy: 0.4959 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4306 - accuracy: 0.4959 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.3034 - accuracy: 0.6250 39/1563 [..............................] - ETA: 2s - loss: 1.3580 - accuracy: 0.5176 77/1563 [>.............................] - ETA: 1s - loss: 1.3508 - accuracy: 0.5292 115/1563 [=>............................] - ETA: 1s - loss: 1.3577 - accuracy: 0.5258 153/1563 [=>............................] - ETA: 1s - loss: 1.3774 - accuracy: 0.5137 191/1563 [==>...........................] - ETA: 1s - loss: 1.3807 - accuracy: 0.5119 229/1563 [===>..........................] - ETA: 1s - loss: 1.3823 - accuracy: 0.5123 267/1563 [====>.........................] - ETA: 1s - loss: 1.3790 - accuracy: 0.5142 305/1563 [====>.........................] - ETA: 1s - loss: 1.3797 - accuracy: 0.5170 343/1563 [=====>........................] - ETA: 1s - loss: 1.3822 - accuracy: 0.5159 381/1563 [======>.......................] - ETA: 1s - loss: 1.3889 - accuracy: 0.5126 419/1563 [=======>......................] - ETA: 1s - loss: 1.3877 - accuracy: 0.5134 457/1563 [=======>......................] - ETA: 1s - loss: 1.3862 - accuracy: 0.5141 495/1563 [========>.....................] - ETA: 1s - loss: 1.3862 - accuracy: 0.5153 533/1563 [=========>....................] - ETA: 1s - loss: 1.3893 - accuracy: 0.5144 571/1563 [=========>....................] - ETA: 1s - loss: 1.3908 - accuracy: 0.5147 609/1563 [==========>...................] - ETA: 1s - loss: 1.3914 - accuracy: 0.5147 647/1563 [===========>..................] - ETA: 1s - loss: 1.3893 - accuracy: 0.5146 685/1563 [============>.................] - ETA: 1s - loss: 1.3880 - accuracy: 0.5152 723/1563 [============>.................] - ETA: 1s - loss: 1.3880 - accuracy: 0.5153 761/1563 [=============>................] - ETA: 1s - loss: 1.3894 - accuracy: 0.5149 799/1563 [==============>...............] - ETA: 1s - loss: 1.3884 - accuracy: 0.5150 837/1563 [===============>..............] - ETA: 0s - loss: 1.3908 - accuracy: 0.5143 875/1563 [===============>..............] - ETA: 0s - loss: 1.3906 - accuracy: 0.5140 913/1563 [================>.............] - ETA: 0s - loss: 1.3908 - accuracy: 0.5140 951/1563 [=================>............] - ETA: 0s - loss: 1.3896 - accuracy: 0.5146 989/1563 [=================>............] - ETA: 0s - loss: 1.3893 - accuracy: 0.5143 1026/1563 [==================>...........] - ETA: 0s - loss: 1.3895 - accuracy: 0.5140 1064/1563 [===================>..........] - ETA: 0s - loss: 1.3883 - accuracy: 0.5136 1102/1563 [====================>.........] - ETA: 0s - loss: 1.3891 - accuracy: 0.5133 1140/1563 [====================>.........] - ETA: 0s - loss: 1.3894 - accuracy: 0.5132 1177/1563 [=====================>........] - ETA: 0s - loss: 1.3884 - accuracy: 0.5140 1214/1563 [======================>.......] - ETA: 0s - loss: 1.3869 - accuracy: 0.5144 1252/1563 [=======================>......] - ETA: 0s - loss: 1.3872 - accuracy: 0.5142 1290/1563 [=======================>......] - ETA: 0s - loss: 1.3867 - accuracy: 0.5141 1328/1563 [========================>.....] - ETA: 0s - loss: 1.3861 - accuracy: 0.5142 1366/1563 [=========================>....] - ETA: 0s - loss: 1.3865 - accuracy: 0.5137 1404/1563 [=========================>....] - ETA: 0s - loss: 1.3858 - accuracy: 0.5139 1442/1563 [==========================>...] - ETA: 0s - loss: 1.3855 - accuracy: 0.5140 1480/1563 [===========================>..] - ETA: 0s - loss: 1.3870 - accuracy: 0.5136 1518/1563 [============================>.] - ETA: 0s - loss: 1.3876 - accuracy: 0.5132 1556/1563 [============================>.] - ETA: 0s - loss: 1.3867 - accuracy: 0.5136 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3874 - accuracy: 0.5131 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.3519 - accuracy: 0.5312 39/1563 [..............................] - ETA: 2s - loss: 1.3452 - accuracy: 0.5280 77/1563 [>.............................] - ETA: 1s - loss: 1.3247 - accuracy: 0.5341 115/1563 [=>............................] - ETA: 1s - loss: 1.3397 - accuracy: 0.5304 153/1563 [=>............................] - ETA: 1s - loss: 1.3388 - accuracy: 0.5304 191/1563 [==>...........................] - ETA: 1s - loss: 1.3446 - accuracy: 0.5286 229/1563 [===>..........................] - ETA: 1s - loss: 1.3407 - accuracy: 0.5273 266/1563 [====>.........................] - ETA: 1s - loss: 1.3428 - accuracy: 0.5261 304/1563 [====>.........................] - ETA: 1s - loss: 1.3372 - accuracy: 0.5306 342/1563 [=====>........................] - ETA: 1s - loss: 1.3411 - accuracy: 0.5292 380/1563 [======>.......................] - ETA: 1s - loss: 1.3411 - accuracy: 0.5275 418/1563 [=======>......................] - ETA: 1s - loss: 1.3451 - accuracy: 0.5244 456/1563 [=======>......................] - ETA: 1s - loss: 1.3427 - accuracy: 0.5259 494/1563 [========>.....................] - ETA: 1s - loss: 1.3441 - accuracy: 0.5276 532/1563 [=========>....................] - ETA: 1s - loss: 1.3429 - accuracy: 0.5294 569/1563 [=========>....................] - ETA: 1s - loss: 1.3412 - accuracy: 0.5311 607/1563 [==========>...................] - ETA: 1s - loss: 1.3445 - accuracy: 0.5301 645/1563 [===========>..................] - ETA: 1s - loss: 1.3432 - accuracy: 0.5313 683/1563 [============>.................] - ETA: 1s - loss: 1.3446 - accuracy: 0.5300 721/1563 [============>.................] - ETA: 1s - loss: 1.3452 - accuracy: 0.5303 759/1563 [=============>................] - ETA: 1s - loss: 1.3448 - accuracy: 0.5302 797/1563 [==============>...............] - ETA: 1s - loss: 1.3433 - accuracy: 0.5300 835/1563 [===============>..............] - ETA: 0s - loss: 1.3456 - accuracy: 0.5285 873/1563 [===============>..............] - ETA: 0s - loss: 1.3465 - accuracy: 0.5275 911/1563 [================>.............] - ETA: 0s - loss: 1.3472 - accuracy: 0.5270 949/1563 [=================>............] - ETA: 0s - loss: 1.3488 - accuracy: 0.5265 987/1563 [=================>............] - ETA: 0s - loss: 1.3488 - accuracy: 0.5271 1025/1563 [==================>...........] - ETA: 0s - loss: 1.3506 - accuracy: 0.5265 1063/1563 [===================>..........] - ETA: 0s - loss: 1.3514 - accuracy: 0.5262 1101/1563 [====================>.........] - ETA: 0s - loss: 1.3511 - accuracy: 0.5262 1139/1563 [====================>.........] - ETA: 0s - loss: 1.3517 - accuracy: 0.5263 1176/1563 [=====================>........] - ETA: 0s - loss: 1.3511 - accuracy: 0.5265 1214/1563 [======================>.......] - ETA: 0s - loss: 1.3498 - accuracy: 0.5275 1251/1563 [=======================>......] - ETA: 0s - loss: 1.3519 - accuracy: 0.5266 1289/1563 [=======================>......] - ETA: 0s - loss: 1.3499 - accuracy: 0.5272 1327/1563 [========================>.....] - ETA: 0s - loss: 1.3499 - accuracy: 0.5267 1365/1563 [=========================>....] - ETA: 0s - loss: 1.3504 - accuracy: 0.5268 1403/1563 [=========================>....] - ETA: 0s - loss: 1.3491 - accuracy: 0.5272 1441/1563 [==========================>...] - ETA: 0s - loss: 1.3490 - accuracy: 0.5275 1479/1563 [===========================>..] - ETA: 0s - loss: 1.3493 - accuracy: 0.5270 1517/1563 [============================>.] - ETA: 0s - loss: 1.3486 - accuracy: 0.5275 1554/1563 [============================>.] - ETA: 0s - loss: 1.3485 - accuracy: 0.5273 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3483 - accuracy: 0.5274 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.2402 - accuracy: 0.5000 39/1563 [..............................] - ETA: 2s - loss: 1.3210 - accuracy: 0.5312 77/1563 [>.............................] - ETA: 1s - loss: 1.2964 - accuracy: 0.5450 115/1563 [=>............................] - ETA: 1s - loss: 1.3006 - accuracy: 0.5386 154/1563 [=>............................] - ETA: 1s - loss: 1.2983 - accuracy: 0.5384 192/1563 [==>...........................] - ETA: 1s - loss: 1.2912 - accuracy: 0.5420 230/1563 [===>..........................] - ETA: 1s - loss: 1.2989 - accuracy: 0.5410 268/1563 [====>.........................] - ETA: 1s - loss: 1.2937 - accuracy: 0.5444 306/1563 [====>.........................] - ETA: 1s - loss: 1.2937 - accuracy: 0.5445 343/1563 [=====>........................] - ETA: 1s - loss: 1.2995 - accuracy: 0.5414 381/1563 [======>.......................] - ETA: 1s - loss: 1.3028 - accuracy: 0.5400 419/1563 [=======>......................] - ETA: 1s - loss: 1.3046 - accuracy: 0.5399 457/1563 [=======>......................] - ETA: 1s - loss: 1.3066 - accuracy: 0.5403 494/1563 [========>.....................] - ETA: 1s - loss: 1.3082 - accuracy: 0.5399 532/1563 [=========>....................] - ETA: 1s - loss: 1.3096 - accuracy: 0.5402 570/1563 [=========>....................] - ETA: 1s - loss: 1.3074 - accuracy: 0.5418 608/1563 [==========>...................] - ETA: 1s - loss: 1.3075 - accuracy: 0.5422 646/1563 [===========>..................] - ETA: 1s - loss: 1.3103 - accuracy: 0.5411 684/1563 [============>.................] - ETA: 1s - loss: 1.3121 - accuracy: 0.5407 722/1563 [============>.................] - ETA: 1s - loss: 1.3105 - accuracy: 0.5418 760/1563 [=============>................] - ETA: 1s - loss: 1.3085 - accuracy: 0.5429 798/1563 [==============>...............] - ETA: 1s - loss: 1.3102 - accuracy: 0.5415 836/1563 [===============>..............] - ETA: 0s - loss: 1.3112 - accuracy: 0.5403 874/1563 [===============>..............] - ETA: 0s - loss: 1.3129 - accuracy: 0.5394 912/1563 [================>.............] - ETA: 0s - loss: 1.3121 - accuracy: 0.5400 950/1563 [=================>............] - ETA: 0s - loss: 1.3132 - accuracy: 0.5395 988/1563 [=================>............] - ETA: 0s - loss: 1.3139 - accuracy: 0.5396 1026/1563 [==================>...........] - ETA: 0s - loss: 1.3136 - accuracy: 0.5398 1064/1563 [===================>..........] - ETA: 0s - loss: 1.3127 - accuracy: 0.5401 1102/1563 [====================>.........] - ETA: 0s - loss: 1.3133 - accuracy: 0.5394 1140/1563 [====================>.........] - ETA: 0s - loss: 1.3153 - accuracy: 0.5383 1178/1563 [=====================>........] - ETA: 0s - loss: 1.3131 - accuracy: 0.5392 1216/1563 [======================>.......] - ETA: 0s - loss: 1.3137 - accuracy: 0.5387 1254/1563 [=======================>......] - ETA: 0s - loss: 1.3138 - accuracy: 0.5384 1293/1563 [=======================>......] - ETA: 0s - loss: 1.3139 - accuracy: 0.5384 1331/1563 [========================>.....] - ETA: 0s - loss: 1.3152 - accuracy: 0.5381 1369/1563 [=========================>....] - ETA: 0s - loss: 1.3147 - accuracy: 0.5384 1408/1563 [==========================>...] - ETA: 0s - loss: 1.3145 - accuracy: 0.5382 1446/1563 [==========================>...] - ETA: 0s - loss: 1.3148 - accuracy: 0.5383 1484/1563 [===========================>..] - ETA: 0s - loss: 1.3136 - accuracy: 0.5387 1522/1563 [============================>.] - ETA: 0s - loss: 1.3147 - accuracy: 0.5382 1560/1563 [============================>.] - ETA: 0s - loss: 1.3149 - accuracy: 0.5381 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3148 - accuracy: 0.5382 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.5425 - accuracy: 0.3438 39/1563 [..............................] - ETA: 2s - loss: 1.3132 - accuracy: 0.5401 77/1563 [>.............................] - ETA: 1s - loss: 1.2935 - accuracy: 0.5491 115/1563 [=>............................] - ETA: 1s - loss: 1.2960 - accuracy: 0.5437 153/1563 [=>............................] - ETA: 1s - loss: 1.2994 - accuracy: 0.5433 191/1563 [==>...........................] - ETA: 1s - loss: 1.2953 - accuracy: 0.5452 229/1563 [===>..........................] - ETA: 1s - loss: 1.2946 - accuracy: 0.5437 267/1563 [====>.........................] - ETA: 1s - loss: 1.2879 - accuracy: 0.5459 305/1563 [====>.........................] - ETA: 1s - loss: 1.2860 - accuracy: 0.5444 343/1563 [=====>........................] - ETA: 1s - loss: 1.2925 - accuracy: 0.5419 380/1563 [======>.......................] - ETA: 1s - loss: 1.2912 - accuracy: 0.5431 417/1563 [=======>......................] - ETA: 1s - loss: 1.2886 - accuracy: 0.5447 454/1563 [=======>......................] - ETA: 1s - loss: 1.2906 - accuracy: 0.5441 491/1563 [========>.....................] - ETA: 1s - loss: 1.2911 - accuracy: 0.5435 528/1563 [=========>....................] - ETA: 1s - loss: 1.2889 - accuracy: 0.5451 565/1563 [=========>....................] - ETA: 1s - loss: 1.2882 - accuracy: 0.5459 602/1563 [==========>...................] - ETA: 1s - loss: 1.2892 - accuracy: 0.5453 640/1563 [===========>..................] - ETA: 1s - loss: 1.2898 - accuracy: 0.5456 678/1563 [============>.................] - ETA: 1s - loss: 1.2893 - accuracy: 0.5454 715/1563 [============>.................] - ETA: 1s - loss: 1.2882 - accuracy: 0.5454 752/1563 [=============>................] - ETA: 1s - loss: 1.2867 - accuracy: 0.5464 789/1563 [==============>...............] - ETA: 1s - loss: 1.2883 - accuracy: 0.5456 826/1563 [==============>...............] - ETA: 0s - loss: 1.2885 - accuracy: 0.5452 863/1563 [===============>..............] - ETA: 0s - loss: 1.2882 - accuracy: 0.5463 900/1563 [================>.............] - ETA: 0s - loss: 1.2883 - accuracy: 0.5461 937/1563 [================>.............] - ETA: 0s - loss: 1.2885 - accuracy: 0.5457 974/1563 [=================>............] - ETA: 0s - loss: 1.2865 - accuracy: 0.5467 1012/1563 [==================>...........] - ETA: 0s - loss: 1.2848 - accuracy: 0.5472 1049/1563 [===================>..........] - ETA: 0s - loss: 1.2849 - accuracy: 0.5473 1087/1563 [===================>..........] - ETA: 0s - loss: 1.2855 - accuracy: 0.5471 1124/1563 [====================>.........] - ETA: 0s - loss: 1.2858 - accuracy: 0.5471 1161/1563 [=====================>........] - ETA: 0s - loss: 1.2862 - accuracy: 0.5468 1199/1563 [======================>.......] - ETA: 0s - loss: 1.2850 - accuracy: 0.5477 1237/1563 [======================>.......] - ETA: 0s - loss: 1.2851 - accuracy: 0.5474 1274/1563 [=======================>......] - ETA: 0s - loss: 1.2857 - accuracy: 0.5475 1311/1563 [========================>.....] - ETA: 0s - loss: 1.2859 - accuracy: 0.5474 1348/1563 [========================>.....] - ETA: 0s - loss: 1.2845 - accuracy: 0.5481 1385/1563 [=========================>....] - ETA: 0s - loss: 1.2824 - accuracy: 0.5489 1422/1563 [==========================>...] - ETA: 0s - loss: 1.2840 - accuracy: 0.5480 1459/1563 [===========================>..] - ETA: 0s - loss: 1.2847 - accuracy: 0.5479 1497/1563 [===========================>..] - ETA: 0s - loss: 1.2829 - accuracy: 0.5485 1534/1563 [============================>.] - ETA: 0s - loss: 1.2821 - accuracy: 0.5486 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2818 - accuracy: 0.5490 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.2327 - accuracy: 0.5000 38/1563 [..............................] - ETA: 2s - loss: 1.2427 - accuracy: 0.5518 76/1563 [>.............................] - ETA: 2s - loss: 1.2278 - accuracy: 0.5576 113/1563 [=>............................] - ETA: 1s - loss: 1.2328 - accuracy: 0.5647 150/1563 [=>............................] - ETA: 1s - loss: 1.2284 - accuracy: 0.5685 187/1563 [==>...........................] - ETA: 1s - loss: 1.2282 - accuracy: 0.5653 224/1563 [===>..........................] - ETA: 1s - loss: 1.2344 - accuracy: 0.5640 261/1563 [====>.........................] - ETA: 1s - loss: 1.2358 - accuracy: 0.5629 298/1563 [====>.........................] - ETA: 1s - loss: 1.2382 - accuracy: 0.5630 336/1563 [=====>........................] - ETA: 1s - loss: 1.2375 - accuracy: 0.5636 373/1563 [======>.......................] - ETA: 1s - loss: 1.2393 - accuracy: 0.5631 410/1563 [======>.......................] - ETA: 1s - loss: 1.2408 - accuracy: 0.5626 447/1563 [=======>......................] - ETA: 1s - loss: 1.2432 - accuracy: 0.5647 484/1563 [========>.....................] - ETA: 1s - loss: 1.2444 - accuracy: 0.5628 521/1563 [=========>....................] - ETA: 1s - loss: 1.2437 - accuracy: 0.5629 558/1563 [=========>....................] - ETA: 1s - loss: 1.2424 - accuracy: 0.5640 595/1563 [==========>...................] - ETA: 1s - loss: 1.2423 - accuracy: 0.5637 632/1563 [===========>..................] - ETA: 1s - loss: 1.2453 - accuracy: 0.5632 669/1563 [===========>..................] - ETA: 1s - loss: 1.2460 - accuracy: 0.5635 706/1563 [============>.................] - ETA: 1s - loss: 1.2453 - accuracy: 0.5632 743/1563 [=============>................] - ETA: 1s - loss: 1.2452 - accuracy: 0.5634 781/1563 [=============>................] - ETA: 1s - loss: 1.2439 - accuracy: 0.5636 819/1563 [==============>...............] - ETA: 1s - loss: 1.2422 - accuracy: 0.5636 856/1563 [===============>..............] - ETA: 0s - loss: 1.2436 - accuracy: 0.5639 894/1563 [================>.............] - ETA: 0s - loss: 1.2472 - accuracy: 0.5628 932/1563 [================>.............] - ETA: 0s - loss: 1.2467 - accuracy: 0.5629 970/1563 [=================>............] - ETA: 0s - loss: 1.2476 - accuracy: 0.5626 1007/1563 [==================>...........] - ETA: 0s - loss: 1.2489 - accuracy: 0.5623 1044/1563 [===================>..........] - ETA: 0s - loss: 1.2500 - accuracy: 0.5623 1082/1563 [===================>..........] - ETA: 0s - loss: 1.2508 - accuracy: 0.5619 1120/1563 [====================>.........] - ETA: 0s - loss: 1.2498 - accuracy: 0.5621 1157/1563 [=====================>........] - ETA: 0s - loss: 1.2503 - accuracy: 0.5616 1194/1563 [=====================>........] - ETA: 0s - loss: 1.2504 - accuracy: 0.5614 1231/1563 [======================>.......] - ETA: 0s - loss: 1.2510 - accuracy: 0.5610 1268/1563 [=======================>......] - ETA: 0s - loss: 1.2516 - accuracy: 0.5605 1305/1563 [========================>.....] - ETA: 0s - loss: 1.2516 - accuracy: 0.5603 1342/1563 [========================>.....] - ETA: 0s - loss: 1.2524 - accuracy: 0.5600 1379/1563 [=========================>....] - ETA: 0s - loss: 1.2527 - accuracy: 0.5598 1417/1563 [==========================>...] - ETA: 0s - loss: 1.2525 - accuracy: 0.5601 1455/1563 [==========================>...] - ETA: 0s - loss: 1.2535 - accuracy: 0.5599 1492/1563 [===========================>..] - ETA: 0s - loss: 1.2540 - accuracy: 0.5597 1529/1563 [============================>.] - ETA: 0s - loss: 1.2532 - accuracy: 0.5600 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2532 - accuracy: 0.5602 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 23.855648279190063 hydro04 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins7810029610055880196.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS