![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
Started by user Jeremy Enos Running as SYSTEM Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace [SSH] script: module load anaconda3_gpu module load cudnn cd python rm -f tensorflow_test.txt srun --nodes=1 --ntasks-per-node=1 --mem=16g \ --time=00:05:00 \ --partition=a100rome \ --account=bbmb-hydro \ --gpus-per-node=1 \ --gpu-bind=closest \ python3 cifar10gpu.py | tee tensorflow_test.txt YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2` echo YVALUE=$YVALUE > tensorflow_time.txt [SSH] executing... srun: job 96826 queued and waiting for resources srun: job 96826 has been allocated resources 2024-11-06 09:46:26.805018: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-06 09:46:50.398066: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-06 09:47:21.841861: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-06 09:47:21.842461: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib 2024-11-06 09:47:21.842470: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2024-11-06 09:48:09.230151: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-06 09:48:11.523048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0 (50000, 32, 32, 3) (10000, 32, 32, 3) Epoch 1/10 2024-11-06 09:48:19.499998: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 1/1563 [..............................] - ETA: 1:55:55 - loss: 2.4229 - accuracy: 0.1250 36/1563 [..............................] - ETA: 2s - loss: 2.2768 - accuracy: 0.1571 74/1563 [>.............................] - ETA: 2s - loss: 2.1848 - accuracy: 0.2027 112/1563 [=>............................] - ETA: 1s - loss: 2.1384 - accuracy: 0.2160 150/1563 [=>............................] - ETA: 1s - loss: 2.1034 - accuracy: 0.2300 188/1563 [==>...........................] - ETA: 1s - loss: 2.0719 - accuracy: 0.2455 226/1563 [===>..........................] - ETA: 1s - loss: 2.0460 - accuracy: 0.2544 264/1563 [====>.........................] - ETA: 1s - loss: 2.0216 - accuracy: 0.2642 302/1563 [====>.........................] - ETA: 1s - loss: 2.0042 - accuracy: 0.2709 340/1563 [=====>........................] - ETA: 1s - loss: 1.9945 - accuracy: 0.2752 378/1563 [======>.......................] - ETA: 1s - loss: 1.9808 - accuracy: 0.2825 416/1563 [======>.......................] - ETA: 1s - loss: 1.9710 - accuracy: 0.2874 454/1563 [=======>......................] - ETA: 1s - loss: 1.9592 - accuracy: 0.2927 492/1563 [========>.....................] - ETA: 1s - loss: 1.9457 - accuracy: 0.2981 531/1563 [=========>....................] - ETA: 1s - loss: 1.9339 - accuracy: 0.3051 569/1563 [=========>....................] - ETA: 1s - loss: 1.9258 - accuracy: 0.3078 607/1563 [==========>...................] - ETA: 1s - loss: 1.9175 - accuracy: 0.3118 645/1563 [===========>..................] - ETA: 1s - loss: 1.9115 - accuracy: 0.3141 683/1563 [============>.................] - ETA: 1s - loss: 1.9037 - accuracy: 0.3178 721/1563 [============>.................] - ETA: 1s - loss: 1.8961 - accuracy: 0.3209 759/1563 [=============>................] - ETA: 1s - loss: 1.8888 - accuracy: 0.3234 797/1563 [==============>...............] - ETA: 1s - loss: 1.8834 - accuracy: 0.3255 835/1563 [===============>..............] - ETA: 0s - loss: 1.8776 - accuracy: 0.3276 873/1563 [===============>..............] - ETA: 0s - loss: 1.8724 - accuracy: 0.3293 911/1563 [================>.............] - ETA: 0s - loss: 1.8682 - accuracy: 0.3315 949/1563 [=================>............] - ETA: 0s - loss: 1.8640 - accuracy: 0.3335 987/1563 [=================>............] - ETA: 0s - loss: 1.8613 - accuracy: 0.3342 1025/1563 [==================>...........] - ETA: 0s - loss: 1.8585 - accuracy: 0.3355 1064/1563 [===================>..........] - ETA: 0s - loss: 1.8541 - accuracy: 0.3372 1102/1563 [====================>.........] - ETA: 0s - loss: 1.8499 - accuracy: 0.3389 1140/1563 [====================>.........] - ETA: 0s - loss: 1.8469 - accuracy: 0.3402 1178/1563 [=====================>........] - ETA: 0s - loss: 1.8437 - accuracy: 0.3411 1216/1563 [======================>.......] - ETA: 0s - loss: 1.8396 - accuracy: 0.3427 1254/1563 [=======================>......] - ETA: 0s - loss: 1.8367 - accuracy: 0.3438 1292/1563 [=======================>......] - ETA: 0s - loss: 1.8330 - accuracy: 0.3456 1330/1563 [========================>.....] - ETA: 0s - loss: 1.8303 - accuracy: 0.3468 1368/1563 [=========================>....] - ETA: 0s - loss: 1.8256 - accuracy: 0.3488 1406/1563 [=========================>....] - ETA: 0s - loss: 1.8235 - accuracy: 0.3497 1444/1563 [==========================>...] - ETA: 0s - loss: 1.8205 - accuracy: 0.3512 1483/1563 [===========================>..] - ETA: 0s - loss: 1.8181 - accuracy: 0.3523 1522/1563 [============================>.] - ETA: 0s - loss: 1.8159 - accuracy: 0.3532 1561/1563 [============================>.] - ETA: 0s - loss: 1.8128 - accuracy: 0.3550 1563/1563 [==============================] - 7s 1ms/step - loss: 1.8124 - accuracy: 0.3552 Epoch 2/10 1/1563 [..............................] - ETA: 2s - loss: 1.2937 - accuracy: 0.5312 40/1563 [..............................] - ETA: 2s - loss: 1.7015 - accuracy: 0.3984 79/1563 [>.............................] - ETA: 1s - loss: 1.6953 - accuracy: 0.3936 118/1563 [=>............................] - ETA: 1s - loss: 1.6862 - accuracy: 0.4023 157/1563 [==>...........................] - ETA: 1s - loss: 1.6772 - accuracy: 0.4070 196/1563 [==>...........................] - ETA: 1s - loss: 1.6695 - accuracy: 0.4128 235/1563 [===>..........................] - ETA: 1s - loss: 1.6666 - accuracy: 0.4137 274/1563 [====>.........................] - ETA: 1s - loss: 1.6621 - accuracy: 0.4167 313/1563 [=====>........................] - ETA: 1s - loss: 1.6669 - accuracy: 0.4136 352/1563 [=====>........................] - ETA: 1s - loss: 1.6619 - accuracy: 0.4155 391/1563 [======>.......................] - ETA: 1s - loss: 1.6652 - accuracy: 0.4143 430/1563 [=======>......................] - ETA: 1s - loss: 1.6622 - accuracy: 0.4160 469/1563 [========>.....................] - ETA: 1s - loss: 1.6570 - accuracy: 0.4169 508/1563 [========>.....................] - ETA: 1s - loss: 1.6576 - accuracy: 0.4164 547/1563 [=========>....................] - ETA: 1s - loss: 1.6549 - accuracy: 0.4173 586/1563 [==========>...................] - ETA: 1s - loss: 1.6561 - accuracy: 0.4176 625/1563 [==========>...................] - ETA: 1s - loss: 1.6531 - accuracy: 0.4192 663/1563 [===========>..................] - ETA: 1s - loss: 1.6524 - accuracy: 0.4195 701/1563 [============>.................] - ETA: 1s - loss: 1.6507 - accuracy: 0.4205 739/1563 [=============>................] - ETA: 1s - loss: 1.6490 - accuracy: 0.4211 778/1563 [=============>................] - ETA: 1s - loss: 1.6458 - accuracy: 0.4219 817/1563 [==============>...............] - ETA: 0s - loss: 1.6457 - accuracy: 0.4223 855/1563 [===============>..............] - ETA: 0s - loss: 1.6442 - accuracy: 0.4226 894/1563 [================>.............] - ETA: 0s - loss: 1.6419 - accuracy: 0.4236 933/1563 [================>.............] - ETA: 0s - loss: 1.6404 - accuracy: 0.4231 972/1563 [=================>............] - ETA: 0s - loss: 1.6393 - accuracy: 0.4231 1011/1563 [==================>...........] - ETA: 0s - loss: 1.6387 - accuracy: 0.4236 1050/1563 [===================>..........] - ETA: 0s - loss: 1.6379 - accuracy: 0.4242 1088/1563 [===================>..........] - ETA: 0s - loss: 1.6359 - accuracy: 0.4247 1127/1563 [====================>.........] - ETA: 0s - loss: 1.6348 - accuracy: 0.4249 1166/1563 [=====================>........] - ETA: 0s - loss: 1.6323 - accuracy: 0.4258 1205/1563 [======================>.......] - ETA: 0s - loss: 1.6312 - accuracy: 0.4264 1244/1563 [======================>.......] - ETA: 0s - loss: 1.6295 - accuracy: 0.4270 1283/1563 [=======================>......] - ETA: 0s - loss: 1.6289 - accuracy: 0.4272 1322/1563 [========================>.....] - ETA: 0s - loss: 1.6289 - accuracy: 0.4273 1360/1563 [=========================>....] - ETA: 0s - loss: 1.6282 - accuracy: 0.4277 1399/1563 [=========================>....] - ETA: 0s - loss: 1.6266 - accuracy: 0.4277 1438/1563 [==========================>...] - ETA: 0s - loss: 1.6260 - accuracy: 0.4278 1477/1563 [===========================>..] - ETA: 0s - loss: 1.6251 - accuracy: 0.4277 1515/1563 [============================>.] - ETA: 0s - loss: 1.6250 - accuracy: 0.4279 1553/1563 [============================>.] - ETA: 0s - loss: 1.6243 - accuracy: 0.4280 1563/1563 [==============================] - 2s 1ms/step - loss: 1.6244 - accuracy: 0.4278 Epoch 3/10 1/1563 [..............................] - ETA: 2s - loss: 1.4427 - accuracy: 0.5000 40/1563 [..............................] - ETA: 1s - loss: 1.5373 - accuracy: 0.4500 79/1563 [>.............................] - ETA: 1s - loss: 1.5560 - accuracy: 0.4379 118/1563 [=>............................] - ETA: 1s - loss: 1.5580 - accuracy: 0.4431 157/1563 [==>...........................] - ETA: 1s - loss: 1.5585 - accuracy: 0.4443 196/1563 [==>...........................] - ETA: 1s - loss: 1.5544 - accuracy: 0.4479 235/1563 [===>..........................] - ETA: 1s - loss: 1.5556 - accuracy: 0.4505 273/1563 [====>.........................] - ETA: 1s - loss: 1.5573 - accuracy: 0.4494 312/1563 [====>.........................] - ETA: 1s - loss: 1.5636 - accuracy: 0.4491 351/1563 [=====>........................] - ETA: 1s - loss: 1.5576 - accuracy: 0.4518 390/1563 [======>.......................] - ETA: 1s - loss: 1.5598 - accuracy: 0.4514 429/1563 [=======>......................] - ETA: 1s - loss: 1.5601 - accuracy: 0.4492 468/1563 [=======>......................] - ETA: 1s - loss: 1.5594 - accuracy: 0.4495 507/1563 [========>.....................] - ETA: 1s - loss: 1.5594 - accuracy: 0.4498 545/1563 [=========>....................] - ETA: 1s - loss: 1.5561 - accuracy: 0.4515 584/1563 [==========>...................] - ETA: 1s - loss: 1.5556 - accuracy: 0.4508 622/1563 [==========>...................] - ETA: 1s - loss: 1.5537 - accuracy: 0.4510 660/1563 [===========>..................] - ETA: 1s - loss: 1.5549 - accuracy: 0.4509 699/1563 [============>.................] - ETA: 1s - loss: 1.5538 - accuracy: 0.4516 738/1563 [=============>................] - ETA: 1s - loss: 1.5521 - accuracy: 0.4533 777/1563 [=============>................] - ETA: 1s - loss: 1.5532 - accuracy: 0.4527 815/1563 [==============>...............] - ETA: 0s - loss: 1.5511 - accuracy: 0.4539 854/1563 [===============>..............] - ETA: 0s - loss: 1.5535 - accuracy: 0.4519 893/1563 [================>.............] - ETA: 0s - loss: 1.5511 - accuracy: 0.4535 931/1563 [================>.............] - ETA: 0s - loss: 1.5515 - accuracy: 0.4535 970/1563 [=================>............] - ETA: 0s - loss: 1.5522 - accuracy: 0.4534 1009/1563 [==================>...........] - ETA: 0s - loss: 1.5494 - accuracy: 0.4547 1047/1563 [===================>..........] - ETA: 0s - loss: 1.5473 - accuracy: 0.4549 1086/1563 [===================>..........] - ETA: 0s - loss: 1.5469 - accuracy: 0.4552 1124/1563 [====================>.........] - ETA: 0s - loss: 1.5469 - accuracy: 0.4548 1162/1563 [=====================>........] - ETA: 0s - loss: 1.5470 - accuracy: 0.4550 1201/1563 [======================>.......] - ETA: 0s - loss: 1.5456 - accuracy: 0.4556 1239/1563 [======================>.......] - ETA: 0s - loss: 1.5443 - accuracy: 0.4558 1278/1563 [=======================>......] - ETA: 0s - loss: 1.5436 - accuracy: 0.4561 1317/1563 [========================>.....] - ETA: 0s - loss: 1.5423 - accuracy: 0.4565 1355/1563 [=========================>....] - ETA: 0s - loss: 1.5413 - accuracy: 0.4571 1394/1563 [=========================>....] - ETA: 0s - loss: 1.5413 - accuracy: 0.4568 1433/1563 [==========================>...] - ETA: 0s - loss: 1.5425 - accuracy: 0.4564 1472/1563 [===========================>..] - ETA: 0s - loss: 1.5417 - accuracy: 0.4565 1511/1563 [============================>.] - ETA: 0s - loss: 1.5413 - accuracy: 0.4564 1549/1563 [============================>.] - ETA: 0s - loss: 1.5404 - accuracy: 0.4569 1563/1563 [==============================] - 2s 1ms/step - loss: 1.5402 - accuracy: 0.4570 Epoch 4/10 1/1563 [..............................] - ETA: 1s - loss: 1.5861 - accuracy: 0.4062 40/1563 [..............................] - ETA: 1s - loss: 1.5094 - accuracy: 0.4898 79/1563 [>.............................] - ETA: 1s - loss: 1.4784 - accuracy: 0.4897 118/1563 [=>............................] - ETA: 1s - loss: 1.4823 - accuracy: 0.4876 157/1563 [==>...........................] - ETA: 1s - loss: 1.4804 - accuracy: 0.4918 196/1563 [==>...........................] - ETA: 1s - loss: 1.4836 - accuracy: 0.4887 235/1563 [===>..........................] - ETA: 1s - loss: 1.4790 - accuracy: 0.4894 273/1563 [====>.........................] - ETA: 1s - loss: 1.4782 - accuracy: 0.4881 312/1563 [====>.........................] - ETA: 1s - loss: 1.4739 - accuracy: 0.4891 351/1563 [=====>........................] - ETA: 1s - loss: 1.4767 - accuracy: 0.4876 390/1563 [======>.......................] - ETA: 1s - loss: 1.4825 - accuracy: 0.4852 429/1563 [=======>......................] - ETA: 1s - loss: 1.4896 - accuracy: 0.4823 468/1563 [=======>......................] - ETA: 1s - loss: 1.4871 - accuracy: 0.4828 508/1563 [========>.....................] - ETA: 1s - loss: 1.4867 - accuracy: 0.4811 547/1563 [=========>....................] - ETA: 1s - loss: 1.4852 - accuracy: 0.4800 586/1563 [==========>...................] - ETA: 1s - loss: 1.4882 - accuracy: 0.4791 625/1563 [==========>...................] - ETA: 1s - loss: 1.4877 - accuracy: 0.4803 664/1563 [===========>..................] - ETA: 1s - loss: 1.4861 - accuracy: 0.4803 703/1563 [============>.................] - ETA: 1s - loss: 1.4860 - accuracy: 0.4798 742/1563 [=============>................] - ETA: 1s - loss: 1.4861 - accuracy: 0.4790 781/1563 [=============>................] - ETA: 1s - loss: 1.4881 - accuracy: 0.4780 821/1563 [==============>...............] - ETA: 0s - loss: 1.4882 - accuracy: 0.4781 860/1563 [===============>..............] - ETA: 0s - loss: 1.4889 - accuracy: 0.4775 898/1563 [================>.............] - ETA: 0s - loss: 1.4890 - accuracy: 0.4771 937/1563 [================>.............] - ETA: 0s - loss: 1.4896 - accuracy: 0.4769 976/1563 [=================>............] - ETA: 0s - loss: 1.4886 - accuracy: 0.4773 1015/1563 [==================>...........] - ETA: 0s - loss: 1.4852 - accuracy: 0.4782 1054/1563 [===================>..........] - ETA: 0s - loss: 1.4849 - accuracy: 0.4780 1093/1563 [===================>..........] - ETA: 0s - loss: 1.4836 - accuracy: 0.4784 1132/1563 [====================>.........] - ETA: 0s - loss: 1.4835 - accuracy: 0.4784 1171/1563 [=====================>........] - ETA: 0s - loss: 1.4857 - accuracy: 0.4772 1210/1563 [======================>.......] - ETA: 0s - loss: 1.4862 - accuracy: 0.4767 1248/1563 [======================>.......] - ETA: 0s - loss: 1.4867 - accuracy: 0.4761 1287/1563 [=======================>......] - ETA: 0s - loss: 1.4865 - accuracy: 0.4759 1326/1563 [========================>.....] - ETA: 0s - loss: 1.4853 - accuracy: 0.4764 1365/1563 [=========================>....] - ETA: 0s - loss: 1.4839 - accuracy: 0.4768 1404/1563 [=========================>....] - ETA: 0s - loss: 1.4819 - accuracy: 0.4775 1443/1563 [==========================>...] - ETA: 0s - loss: 1.4805 - accuracy: 0.4778 1482/1563 [===========================>..] - ETA: 0s - loss: 1.4795 - accuracy: 0.4782 1521/1563 [============================>.] - ETA: 0s - loss: 1.4800 - accuracy: 0.4776 1560/1563 [============================>.] - ETA: 0s - loss: 1.4790 - accuracy: 0.4779 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4790 - accuracy: 0.4778 Epoch 5/10 1/1563 [..............................] - ETA: 2s - loss: 1.2562 - accuracy: 0.5625 40/1563 [..............................] - ETA: 1s - loss: 1.4600 - accuracy: 0.4750 79/1563 [>.............................] - ETA: 1s - loss: 1.4584 - accuracy: 0.4703 118/1563 [=>............................] - ETA: 1s - loss: 1.4608 - accuracy: 0.4688 157/1563 [==>...........................] - ETA: 1s - loss: 1.4498 - accuracy: 0.4777 196/1563 [==>...........................] - ETA: 1s - loss: 1.4428 - accuracy: 0.4786 235/1563 [===>..........................] - ETA: 1s - loss: 1.4445 - accuracy: 0.4811 274/1563 [====>.........................] - ETA: 1s - loss: 1.4449 - accuracy: 0.4852 313/1563 [=====>........................] - ETA: 1s - loss: 1.4422 - accuracy: 0.4877 352/1563 [=====>........................] - ETA: 1s - loss: 1.4422 - accuracy: 0.4885 391/1563 [======>.......................] - ETA: 1s - loss: 1.4393 - accuracy: 0.4898 430/1563 [=======>......................] - ETA: 1s - loss: 1.4369 - accuracy: 0.4906 469/1563 [========>.....................] - ETA: 1s - loss: 1.4372 - accuracy: 0.4913 508/1563 [========>.....................] - ETA: 1s - loss: 1.4376 - accuracy: 0.4910 547/1563 [=========>....................] - ETA: 1s - loss: 1.4390 - accuracy: 0.4905 585/1563 [==========>...................] - ETA: 1s - loss: 1.4380 - accuracy: 0.4915 623/1563 [==========>...................] - ETA: 1s - loss: 1.4407 - accuracy: 0.4904 662/1563 [===========>..................] - ETA: 1s - loss: 1.4394 - accuracy: 0.4913 701/1563 [============>.................] - ETA: 1s - loss: 1.4397 - accuracy: 0.4903 740/1563 [=============>................] - ETA: 1s - loss: 1.4377 - accuracy: 0.4914 779/1563 [=============>................] - ETA: 1s - loss: 1.4377 - accuracy: 0.4923 818/1563 [==============>...............] - ETA: 0s - loss: 1.4364 - accuracy: 0.4932 857/1563 [===============>..............] - ETA: 0s - loss: 1.4361 - accuracy: 0.4935 896/1563 [================>.............] - ETA: 0s - loss: 1.4350 - accuracy: 0.4942 935/1563 [================>.............] - ETA: 0s - loss: 1.4363 - accuracy: 0.4935 974/1563 [=================>............] - ETA: 0s - loss: 1.4387 - accuracy: 0.4937 1013/1563 [==================>...........] - ETA: 0s - loss: 1.4387 - accuracy: 0.4942 1052/1563 [===================>..........] - ETA: 0s - loss: 1.4393 - accuracy: 0.4941 1091/1563 [===================>..........] - ETA: 0s - loss: 1.4382 - accuracy: 0.4946 1130/1563 [====================>.........] - ETA: 0s - loss: 1.4374 - accuracy: 0.4950 1169/1563 [=====================>........] - ETA: 0s - loss: 1.4387 - accuracy: 0.4947 1208/1563 [======================>.......] - ETA: 0s - loss: 1.4376 - accuracy: 0.4950 1247/1563 [======================>.......] - ETA: 0s - loss: 1.4367 - accuracy: 0.4954 1286/1563 [=======================>......] - ETA: 0s - loss: 1.4357 - accuracy: 0.4957 1325/1563 [========================>.....] - ETA: 0s - loss: 1.4359 - accuracy: 0.4963 1364/1563 [=========================>....] - ETA: 0s - loss: 1.4345 - accuracy: 0.4969 1403/1563 [=========================>....] - ETA: 0s - loss: 1.4343 - accuracy: 0.4970 1442/1563 [==========================>...] - ETA: 0s - loss: 1.4333 - accuracy: 0.4975 1481/1563 [===========================>..] - ETA: 0s - loss: 1.4332 - accuracy: 0.4978 1520/1563 [============================>.] - ETA: 0s - loss: 1.4325 - accuracy: 0.4976 1557/1563 [============================>.] - ETA: 0s - loss: 1.4312 - accuracy: 0.4978 1563/1563 [==============================] - 2s 1ms/step - loss: 1.4313 - accuracy: 0.4976 Epoch 6/10 1/1563 [..............................] - ETA: 2s - loss: 1.2895 - accuracy: 0.5625 40/1563 [..............................] - ETA: 1s - loss: 1.3856 - accuracy: 0.5031 79/1563 [>.............................] - ETA: 1s - loss: 1.3691 - accuracy: 0.5115 118/1563 [=>............................] - ETA: 1s - loss: 1.3874 - accuracy: 0.5066 157/1563 [==>...........................] - ETA: 1s - loss: 1.3839 - accuracy: 0.5034 195/1563 [==>...........................] - ETA: 1s - loss: 1.3871 - accuracy: 0.5037 234/1563 [===>..........................] - ETA: 1s - loss: 1.4021 - accuracy: 0.5016 273/1563 [====>.........................] - ETA: 1s - loss: 1.3954 - accuracy: 0.5058 311/1563 [====>.........................] - ETA: 1s - loss: 1.3946 - accuracy: 0.5058 349/1563 [=====>........................] - ETA: 1s - loss: 1.3970 - accuracy: 0.5056 388/1563 [======>.......................] - ETA: 1s - loss: 1.3935 - accuracy: 0.5054 427/1563 [=======>......................] - ETA: 1s - loss: 1.3893 - accuracy: 0.5075 466/1563 [=======>......................] - ETA: 1s - loss: 1.3885 - accuracy: 0.5080 505/1563 [========>.....................] - ETA: 1s - loss: 1.3879 - accuracy: 0.5097 544/1563 [=========>....................] - ETA: 1s - loss: 1.3876 - accuracy: 0.5112 583/1563 [==========>...................] - ETA: 1s - loss: 1.3861 - accuracy: 0.5109 622/1563 [==========>...................] - ETA: 1s - loss: 1.3878 - accuracy: 0.5096 660/1563 [===========>..................] - ETA: 1s - loss: 1.3888 - accuracy: 0.5091 699/1563 [============>.................] - ETA: 1s - loss: 1.3897 - accuracy: 0.5096 738/1563 [=============>................] - ETA: 1s - loss: 1.3895 - accuracy: 0.5102 777/1563 [=============>................] - ETA: 1s - loss: 1.3917 - accuracy: 0.5101 816/1563 [==============>...............] - ETA: 0s - loss: 1.3917 - accuracy: 0.5103 855/1563 [===============>..............] - ETA: 0s - loss: 1.3909 - accuracy: 0.5112 894/1563 [================>.............] - ETA: 0s - loss: 1.3914 - accuracy: 0.5112 933/1563 [================>.............] - ETA: 0s - loss: 1.3920 - accuracy: 0.5109 972/1563 [=================>............] - ETA: 0s - loss: 1.3930 - accuracy: 0.5101 1011/1563 [==================>...........] - ETA: 0s - loss: 1.3937 - accuracy: 0.5099 1050/1563 [===================>..........] - ETA: 0s - loss: 1.3934 - accuracy: 0.5096 1089/1563 [===================>..........] - ETA: 0s - loss: 1.3923 - accuracy: 0.5099 1128/1563 [====================>.........] - ETA: 0s - loss: 1.3926 - accuracy: 0.5092 1167/1563 [=====================>........] - ETA: 0s - loss: 1.3929 - accuracy: 0.5092 1205/1563 [======================>.......] - ETA: 0s - loss: 1.3932 - accuracy: 0.5087 1244/1563 [======================>.......] - ETA: 0s - loss: 1.3921 - accuracy: 0.5094 1283/1563 [=======================>......] - ETA: 0s - loss: 1.3918 - accuracy: 0.5096 1322/1563 [========================>.....] - ETA: 0s - loss: 1.3913 - accuracy: 0.5099 1361/1563 [=========================>....] - ETA: 0s - loss: 1.3907 - accuracy: 0.5095 1400/1563 [=========================>....] - ETA: 0s - loss: 1.3894 - accuracy: 0.5104 1439/1563 [==========================>...] - ETA: 0s - loss: 1.3896 - accuracy: 0.5103 1478/1563 [===========================>..] - ETA: 0s - loss: 1.3896 - accuracy: 0.5097 1516/1563 [============================>.] - ETA: 0s - loss: 1.3890 - accuracy: 0.5094 1555/1563 [============================>.] - ETA: 0s - loss: 1.3891 - accuracy: 0.5099 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3890 - accuracy: 0.5101 Epoch 7/10 1/1563 [..............................] - ETA: 2s - loss: 1.5408 - accuracy: 0.3750 40/1563 [..............................] - ETA: 2s - loss: 1.3964 - accuracy: 0.5141 79/1563 [>.............................] - ETA: 1s - loss: 1.3751 - accuracy: 0.5237 118/1563 [=>............................] - ETA: 1s - loss: 1.3603 - accuracy: 0.5328 157/1563 [==>...........................] - ETA: 1s - loss: 1.3585 - accuracy: 0.5297 196/1563 [==>...........................] - ETA: 1s - loss: 1.3549 - accuracy: 0.5297 235/1563 [===>..........................] - ETA: 1s - loss: 1.3614 - accuracy: 0.5270 274/1563 [====>.........................] - ETA: 1s - loss: 1.3668 - accuracy: 0.5247 313/1563 [=====>........................] - ETA: 1s - loss: 1.3642 - accuracy: 0.5251 352/1563 [=====>........................] - ETA: 1s - loss: 1.3652 - accuracy: 0.5242 391/1563 [======>.......................] - ETA: 1s - loss: 1.3613 - accuracy: 0.5248 430/1563 [=======>......................] - ETA: 1s - loss: 1.3613 - accuracy: 0.5263 469/1563 [========>.....................] - ETA: 1s - loss: 1.3551 - accuracy: 0.5287 508/1563 [========>.....................] - ETA: 1s - loss: 1.3526 - accuracy: 0.5284 547/1563 [=========>....................] - ETA: 1s - loss: 1.3559 - accuracy: 0.5274 586/1563 [==========>...................] - ETA: 1s - loss: 1.3566 - accuracy: 0.5273 625/1563 [==========>...................] - ETA: 1s - loss: 1.3562 - accuracy: 0.5268 664/1563 [===========>..................] - ETA: 1s - loss: 1.3557 - accuracy: 0.5266 702/1563 [============>.................] - ETA: 1s - loss: 1.3561 - accuracy: 0.5267 741/1563 [=============>................] - ETA: 1s - loss: 1.3572 - accuracy: 0.5261 780/1563 [=============>................] - ETA: 1s - loss: 1.3564 - accuracy: 0.5248 819/1563 [==============>...............] - ETA: 0s - loss: 1.3533 - accuracy: 0.5255 858/1563 [===============>..............] - ETA: 0s - loss: 1.3558 - accuracy: 0.5249 897/1563 [================>.............] - ETA: 0s - loss: 1.3568 - accuracy: 0.5244 936/1563 [================>.............] - ETA: 0s - loss: 1.3548 - accuracy: 0.5254 975/1563 [=================>............] - ETA: 0s - loss: 1.3533 - accuracy: 0.5260 1014/1563 [==================>...........] - ETA: 0s - loss: 1.3538 - accuracy: 0.5256 1053/1563 [===================>..........] - ETA: 0s - loss: 1.3518 - accuracy: 0.5265 1092/1563 [===================>..........] - ETA: 0s - loss: 1.3539 - accuracy: 0.5255 1131/1563 [====================>.........] - ETA: 0s - loss: 1.3535 - accuracy: 0.5256 1170/1563 [=====================>........] - ETA: 0s - loss: 1.3531 - accuracy: 0.5249 1209/1563 [======================>.......] - ETA: 0s - loss: 1.3532 - accuracy: 0.5247 1248/1563 [======================>.......] - ETA: 0s - loss: 1.3535 - accuracy: 0.5245 1287/1563 [=======================>......] - ETA: 0s - loss: 1.3523 - accuracy: 0.5247 1326/1563 [========================>.....] - ETA: 0s - loss: 1.3525 - accuracy: 0.5247 1365/1563 [=========================>....] - ETA: 0s - loss: 1.3523 - accuracy: 0.5246 1404/1563 [=========================>....] - ETA: 0s - loss: 1.3524 - accuracy: 0.5247 1443/1563 [==========================>...] - ETA: 0s - loss: 1.3526 - accuracy: 0.5246 1481/1563 [===========================>..] - ETA: 0s - loss: 1.3526 - accuracy: 0.5246 1520/1563 [============================>.] - ETA: 0s - loss: 1.3517 - accuracy: 0.5251 1559/1563 [============================>.] - ETA: 0s - loss: 1.3518 - accuracy: 0.5251 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3515 - accuracy: 0.5251 Epoch 8/10 1/1563 [..............................] - ETA: 1s - loss: 1.5190 - accuracy: 0.4688 40/1563 [..............................] - ETA: 1s - loss: 1.2661 - accuracy: 0.5602 79/1563 [>.............................] - ETA: 1s - loss: 1.2762 - accuracy: 0.5510 118/1563 [=>............................] - ETA: 1s - loss: 1.2870 - accuracy: 0.5437 157/1563 [==>...........................] - ETA: 1s - loss: 1.2943 - accuracy: 0.5412 196/1563 [==>...........................] - ETA: 1s - loss: 1.3027 - accuracy: 0.5410 235/1563 [===>..........................] - ETA: 1s - loss: 1.3078 - accuracy: 0.5384 274/1563 [====>.........................] - ETA: 1s - loss: 1.3157 - accuracy: 0.5363 313/1563 [=====>........................] - ETA: 1s - loss: 1.3205 - accuracy: 0.5357 351/1563 [=====>........................] - ETA: 1s - loss: 1.3186 - accuracy: 0.5371 390/1563 [======>.......................] - ETA: 1s - loss: 1.3144 - accuracy: 0.5379 429/1563 [=======>......................] - ETA: 1s - loss: 1.3137 - accuracy: 0.5366 468/1563 [=======>......................] - ETA: 1s - loss: 1.3192 - accuracy: 0.5336 507/1563 [========>.....................] - ETA: 1s - loss: 1.3172 - accuracy: 0.5336 546/1563 [=========>....................] - ETA: 1s - loss: 1.3164 - accuracy: 0.5338 585/1563 [==========>...................] - ETA: 1s - loss: 1.3142 - accuracy: 0.5348 624/1563 [==========>...................] - ETA: 1s - loss: 1.3151 - accuracy: 0.5348 663/1563 [===========>..................] - ETA: 1s - loss: 1.3195 - accuracy: 0.5333 702/1563 [============>.................] - ETA: 1s - loss: 1.3185 - accuracy: 0.5334 741/1563 [=============>................] - ETA: 1s - loss: 1.3206 - accuracy: 0.5334 780/1563 [=============>................] - ETA: 1s - loss: 1.3181 - accuracy: 0.5346 819/1563 [==============>...............] - ETA: 0s - loss: 1.3178 - accuracy: 0.5347 858/1563 [===============>..............] - ETA: 0s - loss: 1.3197 - accuracy: 0.5344 897/1563 [================>.............] - ETA: 0s - loss: 1.3192 - accuracy: 0.5347 936/1563 [================>.............] - ETA: 0s - loss: 1.3194 - accuracy: 0.5346 975/1563 [=================>............] - ETA: 0s - loss: 1.3189 - accuracy: 0.5346 1015/1563 [==================>...........] - ETA: 0s - loss: 1.3170 - accuracy: 0.5355 1054/1563 [===================>..........] - ETA: 0s - loss: 1.3175 - accuracy: 0.5357 1093/1563 [===================>..........] - ETA: 0s - loss: 1.3175 - accuracy: 0.5357 1132/1563 [====================>.........] - ETA: 0s - loss: 1.3177 - accuracy: 0.5354 1171/1563 [=====================>........] - ETA: 0s - loss: 1.3163 - accuracy: 0.5356 1210/1563 [======================>.......] - ETA: 0s - loss: 1.3147 - accuracy: 0.5363 1249/1563 [======================>.......] - ETA: 0s - loss: 1.3128 - accuracy: 0.5367 1288/1563 [=======================>......] - ETA: 0s - loss: 1.3126 - accuracy: 0.5372 1327/1563 [========================>.....] - ETA: 0s - loss: 1.3139 - accuracy: 0.5371 1366/1563 [=========================>....] - ETA: 0s - loss: 1.3143 - accuracy: 0.5367 1405/1563 [=========================>....] - ETA: 0s - loss: 1.3151 - accuracy: 0.5361 1444/1563 [==========================>...] - ETA: 0s - loss: 1.3167 - accuracy: 0.5356 1483/1563 [===========================>..] - ETA: 0s - loss: 1.3155 - accuracy: 0.5355 1522/1563 [============================>.] - ETA: 0s - loss: 1.3151 - accuracy: 0.5357 1561/1563 [============================>.] - ETA: 0s - loss: 1.3153 - accuracy: 0.5359 1563/1563 [==============================] - 2s 1ms/step - loss: 1.3155 - accuracy: 0.5358 Epoch 9/10 1/1563 [..............................] - ETA: 2s - loss: 1.3251 - accuracy: 0.3750 40/1563 [..............................] - ETA: 1s - loss: 1.3173 - accuracy: 0.5172 79/1563 [>.............................] - ETA: 1s - loss: 1.2924 - accuracy: 0.5400 118/1563 [=>............................] - ETA: 1s - loss: 1.2812 - accuracy: 0.5511 157/1563 [==>...........................] - ETA: 1s - loss: 1.2820 - accuracy: 0.5518 195/1563 [==>...........................] - ETA: 1s - loss: 1.2857 - accuracy: 0.5486 233/1563 [===>..........................] - ETA: 1s - loss: 1.2782 - accuracy: 0.5507 272/1563 [====>.........................] - ETA: 1s - loss: 1.2757 - accuracy: 0.5531 311/1563 [====>.........................] - ETA: 1s - loss: 1.2831 - accuracy: 0.5498 349/1563 [=====>........................] - ETA: 1s - loss: 1.2876 - accuracy: 0.5479 387/1563 [======>.......................] - ETA: 1s - loss: 1.2893 - accuracy: 0.5456 426/1563 [=======>......................] - ETA: 1s - loss: 1.2868 - accuracy: 0.5465 464/1563 [=======>......................] - ETA: 1s - loss: 1.2865 - accuracy: 0.5471 502/1563 [========>.....................] - ETA: 1s - loss: 1.2871 - accuracy: 0.5477 540/1563 [=========>....................] - ETA: 1s - loss: 1.2865 - accuracy: 0.5474 578/1563 [==========>...................] - ETA: 1s - loss: 1.2857 - accuracy: 0.5475 617/1563 [==========>...................] - ETA: 1s - loss: 1.2850 - accuracy: 0.5477 656/1563 [===========>..................] - ETA: 1s - loss: 1.2835 - accuracy: 0.5484 695/1563 [============>.................] - ETA: 1s - loss: 1.2855 - accuracy: 0.5481 734/1563 [=============>................] - ETA: 1s - loss: 1.2897 - accuracy: 0.5481 773/1563 [=============>................] - ETA: 1s - loss: 1.2883 - accuracy: 0.5481 811/1563 [==============>...............] - ETA: 0s - loss: 1.2859 - accuracy: 0.5493 849/1563 [===============>..............] - ETA: 0s - loss: 1.2835 - accuracy: 0.5501 887/1563 [================>.............] - ETA: 0s - loss: 1.2832 - accuracy: 0.5508 926/1563 [================>.............] - ETA: 0s - loss: 1.2821 - accuracy: 0.5513 965/1563 [=================>............] - ETA: 0s - loss: 1.2811 - accuracy: 0.5518 1003/1563 [==================>...........] - ETA: 0s - loss: 1.2803 - accuracy: 0.5520 1042/1563 [===================>..........] - ETA: 0s - loss: 1.2821 - accuracy: 0.5513 1080/1563 [===================>..........] - ETA: 0s - loss: 1.2820 - accuracy: 0.5510 1118/1563 [====================>.........] - ETA: 0s - loss: 1.2825 - accuracy: 0.5509 1156/1563 [=====================>........] - ETA: 0s - loss: 1.2832 - accuracy: 0.5504 1195/1563 [=====================>........] - ETA: 0s - loss: 1.2821 - accuracy: 0.5508 1234/1563 [======================>.......] - ETA: 0s - loss: 1.2825 - accuracy: 0.5510 1273/1563 [=======================>......] - ETA: 0s - loss: 1.2828 - accuracy: 0.5505 1312/1563 [========================>.....] - ETA: 0s - loss: 1.2838 - accuracy: 0.5501 1351/1563 [========================>.....] - ETA: 0s - loss: 1.2833 - accuracy: 0.5504 1390/1563 [=========================>....] - ETA: 0s - loss: 1.2847 - accuracy: 0.5499 1428/1563 [==========================>...] - ETA: 0s - loss: 1.2835 - accuracy: 0.5501 1466/1563 [===========================>..] - ETA: 0s - loss: 1.2830 - accuracy: 0.5505 1504/1563 [===========================>..] - ETA: 0s - loss: 1.2837 - accuracy: 0.5500 1542/1563 [============================>.] - ETA: 0s - loss: 1.2837 - accuracy: 0.5498 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2837 - accuracy: 0.5496 Epoch 10/10 1/1563 [..............................] - ETA: 2s - loss: 1.3553 - accuracy: 0.5625 39/1563 [..............................] - ETA: 2s - loss: 1.2034 - accuracy: 0.5761 77/1563 [>.............................] - ETA: 1s - loss: 1.2365 - accuracy: 0.5690 115/1563 [=>............................] - ETA: 1s - loss: 1.2341 - accuracy: 0.5663 153/1563 [=>............................] - ETA: 1s - loss: 1.2519 - accuracy: 0.5637 191/1563 [==>...........................] - ETA: 1s - loss: 1.2592 - accuracy: 0.5582 229/1563 [===>..........................] - ETA: 1s - loss: 1.2511 - accuracy: 0.5598 267/1563 [====>.........................] - ETA: 1s - loss: 1.2530 - accuracy: 0.5595 304/1563 [====>.........................] - ETA: 1s - loss: 1.2490 - accuracy: 0.5597 342/1563 [=====>........................] - ETA: 1s - loss: 1.2475 - accuracy: 0.5601 380/1563 [======>.......................] - ETA: 1s - loss: 1.2467 - accuracy: 0.5600 418/1563 [=======>......................] - ETA: 1s - loss: 1.2493 - accuracy: 0.5593 455/1563 [=======>......................] - ETA: 1s - loss: 1.2468 - accuracy: 0.5615 493/1563 [========>.....................] - ETA: 1s - loss: 1.2475 - accuracy: 0.5607 531/1563 [=========>....................] - ETA: 1s - loss: 1.2475 - accuracy: 0.5613 569/1563 [=========>....................] - ETA: 1s - loss: 1.2459 - accuracy: 0.5623 607/1563 [==========>...................] - ETA: 1s - loss: 1.2463 - accuracy: 0.5611 645/1563 [===========>..................] - ETA: 1s - loss: 1.2444 - accuracy: 0.5621 683/1563 [============>.................] - ETA: 1s - loss: 1.2440 - accuracy: 0.5627 721/1563 [============>.................] - ETA: 1s - loss: 1.2446 - accuracy: 0.5625 759/1563 [=============>................] - ETA: 1s - loss: 1.2469 - accuracy: 0.5613 797/1563 [==============>...............] - ETA: 1s - loss: 1.2500 - accuracy: 0.5606 835/1563 [===============>..............] - ETA: 0s - loss: 1.2494 - accuracy: 0.5610 873/1563 [===============>..............] - ETA: 0s - loss: 1.2518 - accuracy: 0.5599 911/1563 [================>.............] - ETA: 0s - loss: 1.2536 - accuracy: 0.5595 949/1563 [=================>............] - ETA: 0s - loss: 1.2534 - accuracy: 0.5596 987/1563 [=================>............] - ETA: 0s - loss: 1.2517 - accuracy: 0.5596 1025/1563 [==================>...........] - ETA: 0s - loss: 1.2516 - accuracy: 0.5594 1063/1563 [===================>..........] - ETA: 0s - loss: 1.2527 - accuracy: 0.5586 1101/1563 [====================>.........] - ETA: 0s - loss: 1.2534 - accuracy: 0.5582 1139/1563 [====================>.........] - ETA: 0s - loss: 1.2540 - accuracy: 0.5589 1177/1563 [=====================>........] - ETA: 0s - loss: 1.2531 - accuracy: 0.5601 1215/1563 [======================>.......] - ETA: 0s - loss: 1.2517 - accuracy: 0.5602 1253/1563 [=======================>......] - ETA: 0s - loss: 1.2532 - accuracy: 0.5597 1291/1563 [=======================>......] - ETA: 0s - loss: 1.2529 - accuracy: 0.5600 1329/1563 [========================>.....] - ETA: 0s - loss: 1.2533 - accuracy: 0.5597 1368/1563 [=========================>....] - ETA: 0s - loss: 1.2528 - accuracy: 0.5600 1406/1563 [=========================>....] - ETA: 0s - loss: 1.2521 - accuracy: 0.5604 1444/1563 [==========================>...] - ETA: 0s - loss: 1.2510 - accuracy: 0.5607 1482/1563 [===========================>..] - ETA: 0s - loss: 1.2515 - accuracy: 0.5605 1519/1563 [============================>.] - ETA: 0s - loss: 1.2508 - accuracy: 0.5609 1557/1563 [============================>.] - ETA: 0s - loss: 1.2519 - accuracy: 0.5604 1563/1563 [==============================] - 2s 1ms/step - loss: 1.2517 - accuracy: 0.5604 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= flatten (Flatten) (None, 3072) 0 dense (Dense) (None, 3000) 9219000 dense_1 (Dense) (None, 1000) 3001000 dense_2 (Dense) (None, 10) 10010 ================================================================= Total params: 12,230,010 Trainable params: 12,230,010 Non-trainable params: 0 _________________________________________________________________ None wall time seconds (without loading test data set)= 37.026387453079224 hydro03 [SSH] completed [SSH] exit-status: 0 [workspace] $ /bin/sh -xe /tmp/jenkins10757531975915615001.sh + scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace Recording plot data Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt Finished: SUCCESS