Started by timer Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 96867 queued and waiting for resources srun: job 96867 has been allocated resources 2024-11-08 07:43:02.933082: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-08 07:43:03.734592: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-08 07:43:06.831646: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-08 07:43:06.832999: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-08 07:43:06.833008: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-08 07:43:14.174737: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-08 07:43:15.088426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-08 07:43:17.380890: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 29:22 - loss: 2.4344 - accuracy: 0.0938 38/1563 [..............................] - ETA: 2s - loss: 2.2603 - accuracy: 0.1505  76/1563 [>.............................] - ETA: 2s - loss: 2.1924 - accuracy: 0.1850 114/1563 [=>............................] - ETA: 1s - loss: 2.1393 - accuracy: 0.2081 152/1563 [=>............................] - ETA: 1s - loss: 2.1014 - accuracy: 0.2290 191/1563 [==>...........................] - ETA: 1s - loss: 2.0716 - accuracy: 0.2464 230/1563 [===>..........................] - ETA: 1s - loss: 2.0414 - accuracy: 0.2582 268/1563 [====>.........................] - ETA: 1s - loss: 2.0246 - accuracy: 0.2656 307/1563 [====>.........................] - ETA: 1s - loss: 2.0103 - accuracy: 0.2726 345/1563 [=====>........................] - ETA: 1s - loss: 1.9908 - accuracy: 0.2810 383/1563 [======>.......................] - ETA: 1s - loss: 1.9777 - accuracy: 0.2852 422/1563 [=======>......................] - ETA: 1s - loss: 1.9648 - accuracy: 0.2915 460/1563 [=======>......................] - ETA: 1s - loss: 1.9533 - accuracy: 0.2967 498/1563 [========>.....................] - ETA: 1s - loss: 1.9424 - accuracy: 0.3021 536/1563 [=========>....................] - ETA: 1s - loss: 1.9340 - accuracy: 0.3049 573/1563 [=========>....................] - ETA: 1s - loss: 1.9263 - accuracy: 0.3073 610/1563 [==========>...................] - ETA: 1s - loss: 1.9161 - accuracy: 0.3126 648/1563 [===========>..................] - ETA: 1s - loss: 1.9099 - accuracy: 0.3143 686/1563 [============>.................] - ETA: 1s - loss: 1.9025 - accuracy: 0.3181 724/1563 [============>.................] - ETA: 1s - loss: 1.8968 - accuracy: 0.3208 761/1563 [=============>................] - ETA: 1s - loss: 1.8918 - accuracy: 0.3228 799/1563 [==============>...............] - ETA: 1s - loss: 1.8854 - accuracy: 0.3259 837/1563 [===============>..............] - ETA: 0s - loss: 1.8821 - accuracy: 0.3272 875/1563 [===============>..............] - ETA: 0s - loss: 1.8759 - accuracy: 0.3295 913/1563 [================>.............] - ETA: 0s - loss: 1.8721 - accuracy: 0.3313 951/1563 [=================>............] - ETA: 0s - loss: 1.8673 - accuracy: 0.3327 988/1563 [=================>............] - ETA: 0s - loss: 1.8630 - accuracy: 0.3341 1026/1563 [==================>...........] - ETA: 0s - loss: 1.8595 - accuracy: 0.3359 1064/1563 [===================>..........] - ETA: 0s - loss: 1.8549 - accuracy: 0.3373 1101/1563 [====================>.........] - ETA: 0s - loss: 1.8498 - accuracy: 0.3397 1139/1563 [====================>.........] - ETA: 0s - loss: 1.8450 - accuracy: 0.3416 1177/1563 [=====================>........] - ETA: 0s - loss: 1.8417 - accuracy: 0.3428 1215/1563 [======================>.......] - ETA: 0s - loss: 1.8393 - accuracy: 0.3444 1253/1563 [=======================>......] - ETA: 0s - loss: 1.8356 - accuracy: 0.3459 1291/1563 [=======================>......] - ETA: 0s - loss: 1.8312 - accuracy: 0.3472 1329/1563 [========================>.....] - ETA: 0s - loss: 1.8266 - accuracy: 0.3492 1367/1563 [=========================>....] - ETA: 0s - loss: 1.8244 - accuracy: 0.3494 1405/1563 [=========================>....] - ETA: 0s - loss: 1.8213 - accuracy: 0.3506 1443/1563 [==========================>...] - ETA: 0s - loss: 1.8192 - accuracy: 0.3515 1481/1563 [===========================>..] - ETA: 0s - loss: 1.8164 - accuracy: 0.3532 1519/1563 [============================>.] - ETA: 0s - loss: 1.8135 - accuracy: 0.3545 1557/1563 [============================>.] - ETA: 0s - loss: 1.8112 - accuracy: 0.3555 1563/1563 [==============================] - 3s 1ms/step - loss: 1.8108 - accuracy: 0.3557 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.5759 - accuracy: 0.2812 39/1563 [..............................] - ETA: 2s - loss: 1.6326 - accuracy: 0.4111 77/1563 [>.............................] - ETA: 1s - loss: 1.6418 - accuracy: 0.4127 115/1563 [=>............................] - ETA: 1s - loss: 1.6628 - accuracy: 0.4090 153/1563 [=>............................] - ETA: 1s - loss: 1.6605 - accuracy: 0.4134 191/1563 [==>...........................] - ETA: 1s - loss: 1.6608 - accuracy: 0.4097 229/1563 [===>..........................] - ETA: 1s - loss: 1.6642 - accuracy: 0.4091 267/1563 [====>.........................] - ETA: 1s - loss: 1.6658 - accuracy: 0.4102 305/1563 [====>.........................] - ETA: 1s - loss: 1.6660 - accuracy: 0.4108 343/1563 [=====>........................] - ETA: 1s - loss: 1.6694 - accuracy: 0.4112 381/1563 [======>.......................] - ETA: 1s - loss: 1.6651 - accuracy: 0.4117 419/1563 [=======>......................] - ETA: 1s - loss: 1.6648 - accuracy: 0.4113 457/1563 [=======>......................] - ETA: 1s - loss: 1.6640 - accuracy: 0.4099 495/1563 [========>.....................] - ETA: 1s - loss: 1.6608 - accuracy: 0.4105 533/1563 [=========>....................] - ETA: 1s - loss: 1.6577 - accuracy: 0.4111 571/1563 [=========>....................] - ETA: 1s - loss: 1.6552 - accuracy: 0.4111 609/1563 [==========>...................] - ETA: 1s - loss: 1.6513 - accuracy: 0.4136 646/1563 [===========>..................] - ETA: 1s - loss: 1.6498 - accuracy: 0.4145 684/1563 [============>.................] - ETA: 1s - loss: 1.6493 - accuracy: 0.4142 722/1563 [============>.................] - ETA: 1s - loss: 1.6482 - accuracy: 0.4151 760/1563 [=============>................] - ETA: 1s - loss: 1.6464 - accuracy: 0.4156 798/1563 [==============>...............] - ETA: 1s - loss: 1.6435 - accuracy: 0.4161 836/1563 [===============>..............] - ETA: 0s - loss: 1.6433 - accuracy: 0.4168 874/1563 [===============>..............] - ETA: 0s - loss: 1.6422 - accuracy: 0.4175 912/1563 [================>.............] - ETA: 0s - loss: 1.6393 - accuracy: 0.4184 950/1563 [=================>............] - ETA: 0s - loss: 1.6389 - accuracy: 0.4189 987/1563 [=================>............] - ETA: 0s - loss: 1.6377 - accuracy: 0.4203 1024/1563 [==================>...........] - ETA: 0s - loss: 1.6368 - accuracy: 0.4207 1062/1563 [===================>..........] - ETA: 0s - loss: 1.6364 - accuracy: 0.4213 1099/1563 [====================>.........] - ETA: 0s - loss: 1.6359 - accuracy: 0.4212 1137/1563 [====================>.........] - ETA: 0s - loss: 1.6346 - accuracy: 0.4219 1175/1563 [=====================>........] - ETA: 0s - loss: 1.6333 - accuracy: 0.4232 1213/1563 [======================>.......] - ETA: 0s - loss: 1.6326 - accuracy: 0.4235 1251/1563 [=======================>......] - ETA: 0s - loss: 1.6305 - accuracy: 0.4239 1288/1563 [=======================>......] - ETA: 0s - loss: 1.6313 - accuracy: 0.4241 1325/1563 [========================>.....] - ETA: 0s - loss: 1.6319 - accuracy: 0.4238 1363/1563 [=========================>....] - ETA: 0s - loss: 1.6304 - accuracy: 0.4240 1401/1563 [=========================>....] - ETA: 0s - loss: 1.6289 - accuracy: 0.4245 1438/1563 [==========================>...] - ETA: 0s - loss: 1.6280 - accuracy: 0.4251 1475/1563 [===========================>..] - ETA: 0s - loss: 1.6261 - accuracy: 0.4258 1513/1563 [============================>.] - ETA: 0s - loss: 1.6246 - accuracy: 0.4259 1551/1563 [============================>.] - ETA: 0s - loss: 1.6238 - accuracy: 0.4262 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6237 - accuracy: 0.4262 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.7686 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.5215 - accuracy: 0.4527 77/1563 [>.............................] - ETA: 2s - loss: 1.5391 - accuracy: 0.4452 115/1563 [=>............................] - ETA: 1s - loss: 1.5486 - accuracy: 0.4484 153/1563 [=>............................] - ETA: 1s - loss: 1.5647 - accuracy: 0.4434 190/1563 [==>...........................] - ETA: 1s - loss: 1.5543 - accuracy: 0.4502 229/1563 [===>..........................] - ETA: 1s - loss: 1.5557 - accuracy: 0.4499 268/1563 [====>.........................] - ETA: 1s - loss: 1.5533 - accuracy: 0.4509 307/1563 [====>.........................] - ETA: 1s - loss: 1.5540 - accuracy: 0.4512 346/1563 [=====>........................] - ETA: 1s - loss: 1.5522 - accuracy: 0.4523 385/1563 [======>.......................] - ETA: 1s - loss: 1.5526 - accuracy: 0.4512 424/1563 [=======>......................] - ETA: 1s - loss: 1.5561 - accuracy: 0.4506 463/1563 [=======>......................] - ETA: 1s - loss: 1.5522 - accuracy: 0.4519 502/1563 [========>.....................] - ETA: 1s - loss: 1.5572 - accuracy: 0.4496 541/1563 [=========>....................] - ETA: 1s - loss: 1.5593 - accuracy: 0.4496 580/1563 [==========>...................] - ETA: 1s - loss: 1.5566 - accuracy: 0.4524 619/1563 [==========>...................] - ETA: 1s - loss: 1.5567 - accuracy: 0.4526 658/1563 [===========>..................] - ETA: 1s - loss: 1.5577 - accuracy: 0.4517 697/1563 [============>.................] - ETA: 1s - loss: 1.5573 - accuracy: 0.4513 735/1563 [=============>................] - ETA: 1s - loss: 1.5557 - accuracy: 0.4509 774/1563 [=============>................] - ETA: 1s - loss: 1.5537 - accuracy: 0.4510 813/1563 [==============>...............] - ETA: 0s - loss: 1.5513 - accuracy: 0.4518 852/1563 [===============>..............] - ETA: 0s - loss: 1.5488 - accuracy: 0.4529 891/1563 [================>.............] - ETA: 0s - loss: 1.5501 - accuracy: 0.4533 929/1563 [================>.............] - ETA: 0s - loss: 1.5500 - accuracy: 0.4536 968/1563 [=================>............] - ETA: 0s - loss: 1.5506 - accuracy: 0.4539 1007/1563 [==================>...........] - ETA: 0s - loss: 1.5500 - accuracy: 0.4541 1046/1563 [===================>..........] - ETA: 0s - loss: 1.5500 - accuracy: 0.4541 1085/1563 [===================>..........] - ETA: 0s - loss: 1.5500 - accuracy: 0.4544 1124/1563 [====================>.........] - ETA: 0s - loss: 1.5507 - accuracy: 0.4548 1163/1563 [=====================>........] - ETA: 0s - loss: 1.5494 - accuracy: 0.4552 1202/1563 [======================>.......] - ETA: 0s - loss: 1.5490 - accuracy: 0.4554 1241/1563 [======================>.......] - ETA: 0s - loss: 1.5476 - accuracy: 0.4560 1279/1563 [=======================>......] - ETA: 0s - loss: 1.5460 - accuracy: 0.4567 1318/1563 [========================>.....] - ETA: 0s - loss: 1.5463 - accuracy: 0.4572 1356/1563 [=========================>....] - ETA: 0s - loss: 1.5456 - accuracy: 0.4575 1395/1563 [=========================>....] - ETA: 0s - loss: 1.5439 - accuracy: 0.4576 1434/1563 [==========================>...] - ETA: 0s - loss: 1.5425 - accuracy: 0.4583 1473/1563 [===========================>..] - ETA: 0s - loss: 1.5425 - accuracy: 0.4575 1512/1563 [============================>.] - ETA: 0s - loss: 1.5423 - accuracy: 0.4572 1550/1563 [============================>.] - ETA: 0s - loss: 1.5414 - accuracy: 0.4572 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5412 - accuracy: 0.4575 Epoch 4/10 1/1563 [..............................] - ETA: 1s - loss: 1.8522 - accuracy: 0.4062 39/1563 [..............................] - ETA: 2s - loss: 1.5263 - accuracy: 0.4696 78/1563 [>.............................] - ETA: 1s - loss: 1.4818 - accuracy: 0.4800 116/1563 [=>............................] - ETA: 1s - loss: 1.4923 - accuracy: 0.4749 155/1563 [=>............................] - ETA: 1s - loss: 1.4998 - accuracy: 0.4702 193/1563 [==>...........................] - ETA: 1s - loss: 1.5044 - accuracy: 0.4697 231/1563 [===>..........................] - ETA: 1s - loss: 1.5021 - accuracy: 0.4692 269/1563 [====>.........................] - ETA: 1s - loss: 1.5053 - accuracy: 0.4672 307/1563 [====>.........................] - ETA: 1s - loss: 1.5068 - accuracy: 0.4666 345/1563 [=====>........................] - ETA: 1s - loss: 1.5064 - accuracy: 0.4662 383/1563 [======>.......................] - ETA: 1s - loss: 1.5056 - accuracy: 0.4668 422/1563 [=======>......................] - ETA: 1s - loss: 1.5060 - accuracy: 0.4662 461/1563 [=======>......................] - ETA: 1s - loss: 1.5077 - accuracy: 0.4652 500/1563 [========>.....................] - ETA: 1s - loss: 1.5039 - accuracy: 0.4669 539/1563 [=========>....................] - ETA: 1s - loss: 1.5015 - accuracy: 0.4682 577/1563 [==========>...................] - ETA: 1s - loss: 1.4965 - accuracy: 0.4705 616/1563 [==========>...................] - ETA: 1s - loss: 1.4937 - accuracy: 0.4718 654/1563 [===========>..................] - ETA: 1s - loss: 1.4937 - accuracy: 0.4724 692/1563 [============>.................] - ETA: 1s - loss: 1.4928 - accuracy: 0.4737 731/1563 [=============>................] - ETA: 1s - loss: 1.4926 - accuracy: 0.4739 770/1563 [=============>................] - ETA: 1s - loss: 1.4913 - accuracy: 0.4743 809/1563 [==============>...............] - ETA: 0s - loss: 1.4894 - accuracy: 0.4750 847/1563 [===============>..............] - ETA: 0s - loss: 1.4887 - accuracy: 0.4751 885/1563 [===============>..............] - ETA: 0s - loss: 1.4884 - accuracy: 0.4748 923/1563 [================>.............] - ETA: 0s - loss: 1.4882 - accuracy: 0.4757 962/1563 [=================>............] - ETA: 0s - loss: 1.4878 - accuracy: 0.4758 1001/1563 [==================>...........] - ETA: 0s - loss: 1.4869 - accuracy: 0.4764 1040/1563 [==================>...........] - ETA: 0s - loss: 1.4866 - accuracy: 0.4761 1079/1563 [===================>..........] - ETA: 0s - loss: 1.4871 - accuracy: 0.4759 1118/1563 [====================>.........] - ETA: 0s - loss: 1.4880 - accuracy: 0.4757 1157/1563 [=====================>........] - ETA: 0s - loss: 1.4888 - accuracy: 0.4753 1196/1563 [=====================>........] - ETA: 0s - loss: 1.4894 - accuracy: 0.4753 1234/1563 [======================>.......] - ETA: 0s - loss: 1.4877 - accuracy: 0.4756 1273/1563 [=======================>......] - ETA: 0s - loss: 1.4862 - accuracy: 0.4765 1312/1563 [========================>.....] - ETA: 0s - loss: 1.4863 - accuracy: 0.4766 1351/1563 [========================>.....] - ETA: 0s - loss: 1.4867 - accuracy: 0.4769 1390/1563 [=========================>....] - ETA: 0s - loss: 1.4853 - accuracy: 0.4772 1429/1563 [==========================>...] - ETA: 0s - loss: 1.4836 - accuracy: 0.4778 1467/1563 [===========================>..] - ETA: 0s - loss: 1.4835 - accuracy: 0.4777 1505/1563 [===========================>..] - ETA: 0s - loss: 1.4829 - accuracy: 0.4779 1544/1563 [============================>.] - ETA: 0s - loss: 1.4821 - accuracy: 0.4785 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4810 - accuracy: 0.4790 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.5072 - accuracy: 0.5312 40/1563 [..............................] - ETA: 1s - loss: 1.4435 - accuracy: 0.4859 79/1563 [>.............................] - ETA: 1s - loss: 1.4515 - accuracy: 0.4877 118/1563 [=>............................] - ETA: 1s - loss: 1.4567 - accuracy: 0.4833 157/1563 [==>...........................] - ETA: 1s - loss: 1.4382 - accuracy: 0.4896 196/1563 [==>...........................] - ETA: 1s - loss: 1.4444 - accuracy: 0.4888 235/1563 [===>..........................] - ETA: 1s - loss: 1.4448 - accuracy: 0.4878 274/1563 [====>.........................] - ETA: 1s - loss: 1.4427 - accuracy: 0.4880 313/1563 [=====>........................] - ETA: 1s - loss: 1.4439 - accuracy: 0.4890 352/1563 [=====>........................] - ETA: 1s - loss: 1.4390 - accuracy: 0.4919 391/1563 [======>.......................] - ETA: 1s - loss: 1.4330 - accuracy: 0.4940 430/1563 [=======>......................] - ETA: 1s - loss: 1.4303 - accuracy: 0.4954 469/1563 [========>.....................] - ETA: 1s - loss: 1.4279 - accuracy: 0.4976 508/1563 [========>.....................] - ETA: 1s - loss: 1.4275 - accuracy: 0.4964 547/1563 [=========>....................] - ETA: 1s - loss: 1.4296 - accuracy: 0.4949 586/1563 [==========>...................] - ETA: 1s - loss: 1.4329 - accuracy: 0.4934 625/1563 [==========>...................] - ETA: 1s - loss: 1.4329 - accuracy: 0.4936 664/1563 [===========>..................] - ETA: 1s - loss: 1.4337 - accuracy: 0.4931 703/1563 [============>.................] - ETA: 1s - loss: 1.4335 - accuracy: 0.4929 742/1563 [=============>................] - ETA: 1s - loss: 1.4345 - accuracy: 0.4926 781/1563 [=============>................] - ETA: 1s - loss: 1.4362 - accuracy: 0.4914 820/1563 [==============>...............] - ETA: 0s - loss: 1.4370 - accuracy: 0.4905 859/1563 [===============>..............] - ETA: 0s - loss: 1.4359 - accuracy: 0.4913 898/1563 [================>.............] - ETA: 0s - loss: 1.4373 - accuracy: 0.4909 937/1563 [================>.............] - ETA: 0s - loss: 1.4386 - accuracy: 0.4906 976/1563 [=================>............] - ETA: 0s - loss: 1.4378 - accuracy: 0.4911 1015/1563 [==================>...........] - ETA: 0s - loss: 1.4363 - accuracy: 0.4920 1054/1563 [===================>..........] - ETA: 0s - loss: 1.4352 - accuracy: 0.4931 1093/1563 [===================>..........] - ETA: 0s - loss: 1.4353 - accuracy: 0.4938 1131/1563 [====================>.........] - ETA: 0s - loss: 1.4354 - accuracy: 0.4939 1169/1563 [=====================>........] - ETA: 0s - loss: 1.4372 - accuracy: 0.4933 1207/1563 [======================>.......] - ETA: 0s - loss: 1.4363 - accuracy: 0.4938 1244/1563 [======================>.......] - ETA: 0s - loss: 1.4353 - accuracy: 0.4945 1281/1563 [=======================>......] - ETA: 0s - loss: 1.4340 - accuracy: 0.4953 1319/1563 [========================>.....] - ETA: 0s - loss: 1.4327 - accuracy: 0.4953 1356/1563 [=========================>....] - ETA: 0s - loss: 1.4339 - accuracy: 0.4953 1394/1563 [=========================>....] - ETA: 0s - loss: 1.4337 - accuracy: 0.4955 1432/1563 [==========================>...] - ETA: 0s - loss: 1.4333 - accuracy: 0.4961 1470/1563 [===========================>..] - ETA: 0s - loss: 1.4321 - accuracy: 0.4962 1508/1563 [===========================>..] - ETA: 0s - loss: 1.4316 - accuracy: 0.4969 1546/1563 [============================>.] - ETA: 0s - loss: 1.4311 - accuracy: 0.4971 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4309 - accuracy: 0.4970 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.5350 - accuracy: 0.3750 36/1563 [..............................] - ETA: 2s - loss: 1.4164 - accuracy: 0.5078 73/1563 [>.............................] - ETA: 2s - loss: 1.4337 - accuracy: 0.4953 110/1563 [=>............................] - ETA: 2s - loss: 1.4106 - accuracy: 0.5082 147/1563 [=>............................] - ETA: 1s - loss: 1.4088 - accuracy: 0.5091 185/1563 [==>...........................] - ETA: 1s - loss: 1.3969 - accuracy: 0.5130 222/1563 [===>..........................] - ETA: 1s - loss: 1.3969 - accuracy: 0.5094 260/1563 [===>..........................] - ETA: 1s - loss: 1.3968 - accuracy: 0.5097 298/1563 [====>.........................] - ETA: 1s - loss: 1.4002 - accuracy: 0.5065 335/1563 [=====>........................] - ETA: 1s - loss: 1.4007 - accuracy: 0.5071 373/1563 [======>.......................] - ETA: 1s - loss: 1.3996 - accuracy: 0.5077 411/1563 [======>.......................] - ETA: 1s - loss: 1.3969 - accuracy: 0.5103 449/1563 [=======>......................] - ETA: 1s - loss: 1.3941 - accuracy: 0.5107 487/1563 [========>.....................] - ETA: 1s - loss: 1.3911 - accuracy: 0.5116 525/1563 [=========>....................] - ETA: 1s - loss: 1.3883 - accuracy: 0.5116 563/1563 [=========>....................] - ETA: 1s - loss: 1.3885 - accuracy: 0.5125 600/1563 [==========>...................] - ETA: 1s - loss: 1.3895 - accuracy: 0.5121 637/1563 [===========>..................] - ETA: 1s - loss: 1.3883 - accuracy: 0.5124 674/1563 [===========>..................] - ETA: 1s - loss: 1.3881 - accuracy: 0.5117 711/1563 [============>.................] - ETA: 1s - loss: 1.3889 - accuracy: 0.5118 749/1563 [=============>................] - ETA: 1s - loss: 1.3910 - accuracy: 0.5115 787/1563 [==============>...............] - ETA: 1s - loss: 1.3918 - accuracy: 0.5113 825/1563 [==============>...............] - ETA: 0s - loss: 1.3909 - accuracy: 0.5120 863/1563 [===============>..............] - ETA: 0s - loss: 1.3902 - accuracy: 0.5118 901/1563 [================>.............] - ETA: 0s - loss: 1.3862 - accuracy: 0.5125 939/1563 [=================>............] - ETA: 0s - loss: 1.3866 - accuracy: 0.5115 976/1563 [=================>............] - ETA: 0s - loss: 1.3863 - accuracy: 0.5120 1013/1563 [==================>...........] - ETA: 0s - loss: 1.3872 - accuracy: 0.5119 1050/1563 [===================>..........] - ETA: 0s - loss: 1.3866 - accuracy: 0.5123 1088/1563 [===================>..........] - ETA: 0s - loss: 1.3884 - accuracy: 0.5129 1126/1563 [====================>.........] - ETA: 0s - loss: 1.3867 - accuracy: 0.5134 1164/1563 [=====================>........] - ETA: 0s - loss: 1.3865 - accuracy: 0.5136 1202/1563 [======================>.......] - ETA: 0s - loss: 1.3862 - accuracy: 0.5137 1239/1563 [======================>.......] - ETA: 0s - loss: 1.3863 - accuracy: 0.5138 1277/1563 [=======================>......] - ETA: 0s - loss: 1.3863 - accuracy: 0.5138 1314/1563 [========================>.....] - ETA: 0s - loss: 1.3873 - accuracy: 0.5134 1351/1563 [========================>.....] - ETA: 0s - loss: 1.3870 - accuracy: 0.5132 1388/1563 [=========================>....] - ETA: 0s - loss: 1.3868 - accuracy: 0.5134 1426/1563 [==========================>...] - ETA: 0s - loss: 1.3872 - accuracy: 0.5131 1464/1563 [===========================>..] - ETA: 0s - loss: 1.3862 - accuracy: 0.5133 1502/1563 [===========================>..] - ETA: 0s - loss: 1.3865 - accuracy: 0.5129 1539/1563 [============================>.] - ETA: 0s - loss: 1.3871 - accuracy: 0.5127 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3861 - accuracy: 0.5128 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.1675 - accuracy: 0.6250 39/1563 [..............................] - ETA: 2s - loss: 1.3758 - accuracy: 0.5096 78/1563 [>.............................] - ETA: 1s - loss: 1.3607 - accuracy: 0.5244 117/1563 [=>............................] - ETA: 1s - loss: 1.3721 - accuracy: 0.5243 156/1563 [=>............................] - ETA: 1s - loss: 1.3661 - accuracy: 0.5308 194/1563 [==>...........................] - ETA: 1s - loss: 1.3559 - accuracy: 0.5341 233/1563 [===>..........................] - ETA: 1s - loss: 1.3576 - accuracy: 0.5299 268/1563 [====>.........................] - ETA: 1s - loss: 1.3554 - accuracy: 0.5296 306/1563 [====>.........................] - ETA: 1s - loss: 1.3493 - accuracy: 0.5297 343/1563 [=====>........................] - ETA: 1s - loss: 1.3483 - accuracy: 0.5298 380/1563 [======>.......................] - ETA: 1s - loss: 1.3489 - accuracy: 0.5295 418/1563 [=======>......................] - ETA: 1s - loss: 1.3457 - accuracy: 0.5304 456/1563 [=======>......................] - ETA: 1s - loss: 1.3482 - accuracy: 0.5291 494/1563 [========>.....................] - ETA: 1s - loss: 1.3481 - accuracy: 0.5276 532/1563 [=========>....................] - ETA: 1s - loss: 1.3500 - accuracy: 0.5270 570/1563 [=========>....................] - ETA: 1s - loss: 1.3487 - accuracy: 0.5274 608/1563 [==========>...................] - ETA: 1s - loss: 1.3511 - accuracy: 0.5280 645/1563 [===========>..................] - ETA: 1s - loss: 1.3505 - accuracy: 0.5281 682/1563 [============>.................] - ETA: 1s - loss: 1.3524 - accuracy: 0.5278 720/1563 [============>.................] - ETA: 1s - loss: 1.3526 - accuracy: 0.5275 758/1563 [=============>................] - ETA: 1s - loss: 1.3503 - accuracy: 0.5283 796/1563 [==============>...............] - ETA: 1s - loss: 1.3505 - accuracy: 0.5286 834/1563 [===============>..............] - ETA: 0s - loss: 1.3524 - accuracy: 0.5290 871/1563 [===============>..............] - ETA: 0s - loss: 1.3528 - accuracy: 0.5293 908/1563 [================>.............] - ETA: 0s - loss: 1.3549 - accuracy: 0.5283 946/1563 [=================>............] - ETA: 0s - loss: 1.3542 - accuracy: 0.5274 983/1563 [=================>............] - ETA: 0s - loss: 1.3526 - accuracy: 0.5276 1020/1563 [==================>...........] - ETA: 0s - loss: 1.3521 - accuracy: 0.5273 1058/1563 [===================>..........] - ETA: 0s - loss: 1.3532 - accuracy: 0.5258 1097/1563 [====================>.........] - ETA: 0s - loss: 1.3532 - accuracy: 0.5254 1136/1563 [====================>.........] - ETA: 0s - loss: 1.3541 - accuracy: 0.5258 1174/1563 [=====================>........] - ETA: 0s - loss: 1.3529 - accuracy: 0.5259 1210/1563 [======================>.......] - ETA: 0s - loss: 1.3520 - accuracy: 0.5257 1248/1563 [======================>.......] - ETA: 0s - loss: 1.3495 - accuracy: 0.5270 1285/1563 [=======================>......] - ETA: 0s - loss: 1.3503 - accuracy: 0.5271 1322/1563 [========================>.....] - ETA: 0s - loss: 1.3500 - accuracy: 0.5272 1360/1563 [=========================>....] - ETA: 0s - loss: 1.3490 - accuracy: 0.5278 1398/1563 [=========================>....] - ETA: 0s - loss: 1.3489 - accuracy: 0.5282 1436/1563 [==========================>...] - ETA: 0s - loss: 1.3486 - accuracy: 0.5282 1474/1563 [===========================>..] - ETA: 0s - loss: 1.3484 - accuracy: 0.5283 1513/1563 [============================>.] - ETA: 0s - loss: 1.3472 - accuracy: 0.5287 1551/1563 [============================>.] - ETA: 0s - loss: 1.3468 - accuracy: 0.5287 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3473 - accuracy: 0.5285 Epoch 8/10 1/1563 [..............................] - ETA: 2s - loss: 1.6089 - accuracy: 0.3750 39/1563 [..............................] - ETA: 2s - loss: 1.3639 - accuracy: 0.5112 78/1563 [>.............................] - ETA: 1s - loss: 1.3444 - accuracy: 0.5256 117/1563 [=>............................] - ETA: 1s - loss: 1.3263 - accuracy: 0.5366 156/1563 [=>............................] - ETA: 1s - loss: 1.3240 - accuracy: 0.5387 194/1563 [==>...........................] - ETA: 1s - loss: 1.3121 - accuracy: 0.5440 232/1563 [===>..........................] - ETA: 1s - loss: 1.3029 - accuracy: 0.5462 270/1563 [====>.........................] - ETA: 1s - loss: 1.3048 - accuracy: 0.5434 308/1563 [====>.........................] - ETA: 1s - loss: 1.3033 - accuracy: 0.5420 346/1563 [=====>........................] - ETA: 1s - loss: 1.3055 - accuracy: 0.5397 384/1563 [======>.......................] - ETA: 1s - loss: 1.3081 - accuracy: 0.5394 423/1563 [=======>......................] - ETA: 1s - loss: 1.3091 - accuracy: 0.5386 461/1563 [=======>......................] - ETA: 1s - loss: 1.3072 - accuracy: 0.5386 499/1563 [========>.....................] - ETA: 1s - loss: 1.3111 - accuracy: 0.5367 537/1563 [=========>....................] - ETA: 1s - loss: 1.3098 - accuracy: 0.5371 574/1563 [==========>...................] - ETA: 1s - loss: 1.3112 - accuracy: 0.5360 612/1563 [==========>...................] - ETA: 1s - loss: 1.3122 - accuracy: 0.5363 650/1563 [===========>..................] - ETA: 1s - loss: 1.3124 - accuracy: 0.5366 688/1563 [============>.................] - ETA: 1s - loss: 1.3146 - accuracy: 0.5349 727/1563 [============>.................] - ETA: 1s - loss: 1.3149 - accuracy: 0.5347 765/1563 [=============>................] - ETA: 1s - loss: 1.3169 - accuracy: 0.5340 803/1563 [==============>...............] - ETA: 1s - loss: 1.3175 - accuracy: 0.5343 841/1563 [===============>..............] - ETA: 0s - loss: 1.3179 - accuracy: 0.5337 879/1563 [===============>..............] - ETA: 0s - loss: 1.3188 - accuracy: 0.5332 917/1563 [================>.............] - ETA: 0s - loss: 1.3175 - accuracy: 0.5341 955/1563 [=================>............] - ETA: 0s - loss: 1.3201 - accuracy: 0.5339 994/1563 [==================>...........] - ETA: 0s - loss: 1.3199 - accuracy: 0.5343 1033/1563 [==================>...........] - ETA: 0s - loss: 1.3184 - accuracy: 0.5350 1071/1563 [===================>..........] - ETA: 0s - loss: 1.3183 - accuracy: 0.5352 1109/1563 [====================>.........] - ETA: 0s - loss: 1.3197 - accuracy: 0.5348 1147/1563 [=====================>........] - ETA: 0s - loss: 1.3189 - accuracy: 0.5351 1185/1563 [=====================>........] - ETA: 0s - loss: 1.3176 - accuracy: 0.5355 1223/1563 [======================>.......] - ETA: 0s - loss: 1.3170 - accuracy: 0.5359 1261/1563 [=======================>......] - ETA: 0s - loss: 1.3155 - accuracy: 0.5363 1299/1563 [=======================>......] - ETA: 0s - loss: 1.3145 - accuracy: 0.5361 1337/1563 [========================>.....] - ETA: 0s - loss: 1.3153 - accuracy: 0.5360 1375/1563 [=========================>....] - ETA: 0s - loss: 1.3145 - accuracy: 0.5368 1413/1563 [==========================>...] - ETA: 0s - loss: 1.3143 - accuracy: 0.5371 1451/1563 [==========================>...] - ETA: 0s - loss: 1.3145 - accuracy: 0.5374 1489/1563 [===========================>..] - ETA: 0s - loss: 1.3158 - accuracy: 0.5365 1527/1563 [============================>.] - ETA: 0s - loss: 1.3148 - accuracy: 0.5368 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3151 - accuracy: 0.5368 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.0482 - accuracy: 0.6250 40/1563 [..............................] - ETA: 2s - loss: 1.2663 - accuracy: 0.5578 79/1563 [>.............................] - ETA: 1s - loss: 1.2440 - accuracy: 0.5589 117/1563 [=>............................] - ETA: 1s - loss: 1.2469 - accuracy: 0.5588 156/1563 [=>............................] - ETA: 1s - loss: 1.2616 - accuracy: 0.5561 194/1563 [==>...........................] - ETA: 1s - loss: 1.2669 - accuracy: 0.5567 232/1563 [===>..........................] - ETA: 1s - loss: 1.2608 - accuracy: 0.5567 270/1563 [====>.........................] - ETA: 1s - loss: 1.2576 - accuracy: 0.5588 309/1563 [====>.........................] - ETA: 1s - loss: 1.2613 - accuracy: 0.5568 348/1563 [=====>........................] - ETA: 1s - loss: 1.2684 - accuracy: 0.5559 387/1563 [======>.......................] - ETA: 1s - loss: 1.2684 - accuracy: 0.5559 425/1563 [=======>......................] - ETA: 1s - loss: 1.2657 - accuracy: 0.5575 463/1563 [=======>......................] - ETA: 1s - loss: 1.2663 - accuracy: 0.5566 501/1563 [========>.....................] - ETA: 1s - loss: 1.2686 - accuracy: 0.5548 539/1563 [=========>....................] - ETA: 1s - loss: 1.2700 - accuracy: 0.5532 577/1563 [==========>...................] - ETA: 1s - loss: 1.2712 - accuracy: 0.5529 615/1563 [==========>...................] - ETA: 1s - loss: 1.2747 - accuracy: 0.5526 654/1563 [===========>..................] - ETA: 1s - loss: 1.2776 - accuracy: 0.5522 692/1563 [============>.................] - ETA: 1s - loss: 1.2750 - accuracy: 0.5534 730/1563 [=============>................] - ETA: 1s - loss: 1.2726 - accuracy: 0.5540 768/1563 [=============>................] - ETA: 1s - loss: 1.2734 - accuracy: 0.5542 806/1563 [==============>...............] - ETA: 1s - loss: 1.2717 - accuracy: 0.5539 844/1563 [===============>..............] - ETA: 0s - loss: 1.2727 - accuracy: 0.5547 882/1563 [===============>..............] - ETA: 0s - loss: 1.2760 - accuracy: 0.5537 920/1563 [================>.............] - ETA: 0s - loss: 1.2760 - accuracy: 0.5536 958/1563 [=================>............] - ETA: 0s - loss: 1.2783 - accuracy: 0.5523 997/1563 [==================>...........] - ETA: 0s - loss: 1.2799 - accuracy: 0.5522 1036/1563 [==================>...........] - ETA: 0s - loss: 1.2799 - accuracy: 0.5522 1074/1563 [===================>..........] - ETA: 0s - loss: 1.2797 - accuracy: 0.5522 1112/1563 [====================>.........] - ETA: 0s - loss: 1.2824 - accuracy: 0.5510 1150/1563 [=====================>........] - ETA: 0s - loss: 1.2831 - accuracy: 0.5507 1188/1563 [=====================>........] - ETA: 0s - loss: 1.2840 - accuracy: 0.5503 1226/1563 [======================>.......] - ETA: 0s - loss: 1.2847 - accuracy: 0.5500 1264/1563 [=======================>......] - ETA: 0s - loss: 1.2840 - accuracy: 0.5501 1302/1563 [=======================>......] - ETA: 0s - loss: 1.2836 - accuracy: 0.5500 1340/1563 [========================>.....] - ETA: 0s - loss: 1.2828 - accuracy: 0.5502 1378/1563 [=========================>....] - ETA: 0s - loss: 1.2821 - accuracy: 0.5504 1416/1563 [==========================>...] - ETA: 0s - loss: 1.2825 - accuracy: 0.5504 1454/1563 [==========================>...] - ETA: 0s - loss: 1.2832 - accuracy: 0.5500 1492/1563 [===========================>..] - ETA: 0s - loss: 1.2824 - accuracy: 0.5502 1530/1563 [============================>.] - ETA: 0s - loss: 1.2814 - accuracy: 0.5507 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2804 - accuracy: 0.5513 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.3367 - accuracy: 0.5000 39/1563 [..............................] - ETA: 2s - loss: 1.2237 - accuracy: 0.5697 77/1563 [>.............................] - ETA: 1s - loss: 1.2572 - accuracy: 0.5564 115/1563 [=>............................] - ETA: 1s - loss: 1.2518 - accuracy: 0.5652 153/1563 [=>............................] - ETA: 1s - loss: 1.2472 - accuracy: 0.5666 191/1563 [==>...........................] - ETA: 1s - loss: 1.2560 - accuracy: 0.5661 229/1563 [===>..........................] - ETA: 1s - loss: 1.2511 - accuracy: 0.5655 268/1563 [====>.........................] - ETA: 1s - loss: 1.2530 - accuracy: 0.5652 305/1563 [====>.........................] - ETA: 1s - loss: 1.2562 - accuracy: 0.5648 343/1563 [=====>........................] - ETA: 1s - loss: 1.2565 - accuracy: 0.5640 381/1563 [======>.......................] - ETA: 1s - loss: 1.2573 - accuracy: 0.5640 419/1563 [=======>......................] - ETA: 1s - loss: 1.2569 - accuracy: 0.5643 457/1563 [=======>......................] - ETA: 1s - loss: 1.2608 - accuracy: 0.5624 495/1563 [========>.....................] - ETA: 1s - loss: 1.2624 - accuracy: 0.5612 533/1563 [=========>....................] - ETA: 1s - loss: 1.2628 - accuracy: 0.5613 571/1563 [=========>....................] - ETA: 1s - loss: 1.2657 - accuracy: 0.5611 609/1563 [==========>...................] - ETA: 1s - loss: 1.2618 - accuracy: 0.5618 647/1563 [===========>..................] - ETA: 1s - loss: 1.2605 - accuracy: 0.5625 685/1563 [============>.................] - ETA: 1s - loss: 1.2554 - accuracy: 0.5643 723/1563 [============>.................] - ETA: 1s - loss: 1.2553 - accuracy: 0.5637 761/1563 [=============>................] - ETA: 1s - loss: 1.2545 - accuracy: 0.5635 799/1563 [==============>...............] - ETA: 1s - loss: 1.2516 - accuracy: 0.5646 837/1563 [===============>..............] - ETA: 0s - loss: 1.2506 - accuracy: 0.5653 876/1563 [===============>..............] - ETA: 0s - loss: 1.2510 - accuracy: 0.5637 914/1563 [================>.............] - ETA: 0s - loss: 1.2487 - accuracy: 0.5648 952/1563 [=================>............] - ETA: 0s - loss: 1.2470 - accuracy: 0.5650 990/1563 [==================>...........] - ETA: 0s - loss: 1.2478 - accuracy: 0.5638 1029/1563 [==================>...........] - ETA: 0s - loss: 1.2482 - accuracy: 0.5628 1067/1563 [===================>..........] - ETA: 0s - loss: 1.2504 - accuracy: 0.5619 1106/1563 [====================>.........] - ETA: 0s - loss: 1.2511 - accuracy: 0.5616 1144/1563 [====================>.........] - ETA: 0s - loss: 1.2520 - accuracy: 0.5611 1182/1563 [=====================>........] - ETA: 0s - loss: 1.2536 - accuracy: 0.5610 1220/1563 [======================>.......] - ETA: 0s - loss: 1.2532 - accuracy: 0.5610 1258/1563 [=======================>......] - ETA: 0s - loss: 1.2530 - accuracy: 0.5611 1296/1563 [=======================>......] - ETA: 0s - loss: 1.2533 - accuracy: 0.5609 1334/1563 [========================>.....] - ETA: 0s - loss: 1.2527 - accuracy: 0.5614 1372/1563 [=========================>....] - ETA: 0s - loss: 1.2525 - accuracy: 0.5616 1411/1563 [==========================>...] - ETA: 0s - loss: 1.2517 - accuracy: 0.5620 1449/1563 [==========================>...] - ETA: 0s - loss: 1.2510 - accuracy: 0.5618 1488/1563 [===========================>..] - ETA: 0s - loss: 1.2510 - accuracy: 0.5621 1527/1563 [============================>.] - ETA: 0s - loss: 1.2501 - accuracy: 0.5627 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2499 - accuracy: 0.5631 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 24.522833585739136 hydro03 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins14165781174237902901.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS